NIC Teaming/Bonding

ESM Port Assignments

All ports tasked for a specific roll and can not be reallocated

14 ports 1000 Mbps Internal links to Blade Servers

BladeServerSlot1 BladeServerSlot2 BladeServerSlot3 BladeServerSlot4 BladeServerSlot5 BladeServerSlot6 BladeServerSlot7 BladeServerSlot8 BladeServerSlot9 BladeServerSlot10 BladeServerSlot11 BladeServerSlot12 BladeServerSlot13 BladeServerSlot14

INT1 INT2 INT3 INT4 INT5 INT6 INT7 INT8 INT9 INT10 INT11 INT12 INT13 INT14 Mgt15 Mgt16

EXT1 EXT2 EXT3 EXT4

4 or 6 ports 10/100/1000 Mbps RJ45 or 4 ports GigE SFP Fiber for external network connections

Serial

Service port

ESM Module

2 ports -100 Mbps MM1 Internal links to the MM2 Management Modules

ESM Connections BladeCenter

BladeCenter Ethernet Port Assignments

Top ESM (Bay 1) 17 18 19 20 ESM External uplinks

9 10 11 12 13 14

15

16

Eth1

1 B S S 1

1 B S S 2

1 B S S 3

1 B S S 4

1 B S S 5

1 B S S 6

1 B S S 7

1 B S S 8

1 B S S 9

1 B S S 1 0

1 B S S 1 1

1 B S S 1 2

Eth0

MM1 Uplink

B B S S S S 1 1 3 4

MM1

MM2
Eth0

2
Eth1

MM2 Uplink

9 10 11 12 13 14

15

16

Bottom ESM (Bay 2)

17 18 19 20

ESM External uplinks

BladeCenter Chassis

(BSS = Blade Server Slot)

Broadcom BASP
Broadcom Advanced Control Suite (BACS) is the GUI tool for BASP.
Windows Only

Broadcom offers no support for Linux and is now ending development. Broadcom now supports the TG3 drivers for Linux
http://www.broadcom.com/support/ethernet_nic/faq_drivers.php#tg3

IBM is distributing and supporting BASP???

What can Broadcom do?

Create a team using: Smart Load Balancing, IEEE 802.3ad (LACP) or Generic Trunking 802.3ad-draft (Cisco EtherChannel). Create VLAN using 802.1q (max. 64). Both of the above can be done concurrently (team w/ VLANs) Enable 802.1p (Quality of Service) New adapters are TCP Offload Engine (TOE) Enabled.
iSCSI and RDMA Capable Included on planar board on HS21, LS21, LS41 (5708s, 5706s) Requires Microsoft Scalable Networking Pack
http://www.microsoft.com/technet/itsolutions/network/snp/default.mspx

BASP Teaming can be used with third-party NICs except for VLAN support.
http://www.broadcom.com/collateral/wp/570X-WP100-R.pdf
5

Flow Control IEEE 802.3x

Flow Control is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames. PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting. The Server link is 1000 Mbps, Full-Duplex, Flow Control Rx/Tx. -> You cant change it!!! Flow Control is especially important when the uplinks are using 100Mbps, since all servers are at 1000 Mbps. When the switch buffer memory is getting full, the GbESM issue a PAUSE frame to the server (hold-on for a moment). To see the link status: /info/link

Smart Load Balancing (SLB)

Proprietary solution from Broadcom. Only works in a flat Layer 2 environment (no routers). Team MAC address is one of the physical MACs of the NICs in the team. Its used to initially reply to ARP requests. Transmit Load Balancing.
When the server receives an ARP request, it replies using the Team MAC. Based on the TCP/UDP information; the teaming driver will change the Source MAC address with one of the MACs of the physical NICs.

Receive Load Balancing

In order to distribute all incoming traffic among all NICs, BASP periodically issues a Gratuitous ARP (GARP) directly to the destination MAC address with the Source MAC address of one of the physical NICs. Router will receive this GARP and keep updating the ARP cache. ARP doesn't have an IP header, so it cant be routed.

Smart Load Balancing (SLB)

Always configure SLB in Active-Standby mode. Make a note of the physical MAC address of each NIC before creating the Team. Once a NIC is part of a Team, it cant be used in another Team. SLB with auto Fall-Back means that once the primary NIC is back on-line, the Team will start using that NIC again.
Keep in mind that when you replace a faulty switch, it doesn't have any configuration, so it might not be good to automatically use it. The MM also perform a series of Link UP/Down tests during the boot process. SLB with no failback is preferable for these reasons.

Generic Trunking
Equivalent of Cisco EtherChannel or Nortel Static MLT. Team uses one MAC and IP address. Requires both links to end on the same device also configured with the same Trunking protocol. Only possible in a BladeCenter using the HS40 or LS41 or the BCE card. If the NICs are connected to different devices, Active-Standby is the only possible option If both NICs in the team are in ActiveActive, the external GbESM ports need to end on a single switch where all ports use same trunking protocol.
Core Network

Generic Trunk

Active

Active

Linux Bonding (802.3ad)

Functionally very similar to what BASP provides. The bond uses the MAC address of the first physical NIC. Output traffic is allocated to physical NICs based on (src hw addr XOR dst hw addr). This selects the same NIC for traffic to a specific destination hw address. SLES Bonding
https://secure-support.novell.com/KanisaPlatform/Publishing/133/3929220_f.SAL_Public.html

Red Hat Bonding

http://kbase.redhat.com/faq/FAQ_85_6313.shtm

VMWare ESX Bonding

http://www.vmware.com/vmtn/resources/432

Link to Linux Networking Stuff.

10

Linux Bonding config in Active-Backup

Bond need to be mode=1 (mode=0 is Active-Active). The NIC in backup is the one with the NOARP statement. By default, preempt is enable. -> Fallback to the original NIC once its OK.
sys1# sys1# sys1# sys1# sys1# bond0 insmod bonding.o mode=1 miimon=100 /sbin/ifconfig bond0 10.0.0.1 up /sbin/ifenslave bond0 eth0 /sbin/ifenslave bond0 eth1 /sbin/ifconfig Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 TX packets:3286647 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.00.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643167 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:10 Base address:0x1080 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING SLAVE NOARP MULTICAST MTU:1500 Metric:1 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:9 Base address:0x1400

eth0

eth1

11

Locating BASP software

Software and documentation is included in driver package (.zip or .iso packages)
NetXtreme (xseries servers)
http://www-3.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-43815

NetXtreme II (newer, system x servers)

http://www-3.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-64505

Default extraction location is C:\Drivers\v94B2

Click on Launch.exe file

12

Installing BASP
This application allows you to update the Windows Drivers (recommended) Management Applications is the BASP Software Default installation is acceptable

13

BACS Control Panel

Tray icon Unconfigured Control Panel

14

Creating a Team
Expert Mode Wizard Mode Be careful of Team Name, as this becomes the Interface Name
Can cause problems if name has a space

15

Configuring BASP
Choose the Team type you wish to configure SLB is most common

16

Add adapters to Team

17

Configuring BASP
ALWAYS configure the SLB Team with one NIC Active and One NIC Standby. Make sure to properly identify the Active and Standby NIC, its not always the one we think... Smart Load Balancing (SLB) is supported with or without FallBack. SLB Auto-Fallback means the Team will go automaticaly to the primary NIC when is comes (back) online.

Note: Before a team is configured on the server ipconfig /all will list the NIC MAC address. On the switch /i/l2/fdb/dump will show you the MAC database.

18

LiveLink
Do not configure Livelink Trunk Monitoring performs this function

19

Add VLAN
Choose whether or not to create VLANs for the TEAM VLAN tagging not supported with non-broadcom adapters

20

Give Name to VLAN

Friendly name for VLAN Not the actual VLAN ID

21

To Tag, or Not To Tag

Only one untagged VLAN per team. GbESM will handle these frames in the PVID

22

Assign the VLAN tag value

This is the actual VLAN id. VLAN must be configured on GbESM Must assign IP address for the adapter (static or DHCP)

23

A chance to review or commit configuration

Almost complete

24

Preview
Shows how configuration will look prior to saving. Make any changes in this view. (dont click cancel thinking you will return to the wizard)

25

Changes to interfaces in the OS

26

High-Availability at the external Network level

NICs on server blades operate in Teaming mode. The NIC Teaming option is unaware of any outages on the external network When a failure occurs on the uplinks, the NIC Teaming function is not aware of the failure and the servers continue to send data to a dead end path! All data is dropped by the primary switch. No error message is sent to the administrator Although two switches are used in the BladeCenter the application sees no resilience to failures
27

Core Network

Layer 2 switches

Active

Backup

High-Availability at the external Network level

Core Network

Active

Backup

With the GbESM Trunk Monitoring option, the switch constantly monitors the uplinks. When a failure is detected the GbESM disables the internal ports. The NIC Teaming function then becomes aware of the failure and falls back to the secondary NIC. A Syslog message and SNMP trap are generated and sent to the Network administrator.

28

Enhanced L2 Trunk Fail-Over

Core Network

Wait until Spanning-Tree is Forwarding.

GbESM is VLAN aware and will only disable internal ports associated with the same VLAN. No un-necessary Fail-Over. Triggers can be configured to force a Fail-Over when only some of the links fail, for bandwidth-sensitive applications. Link Aggregation with Fail-Over can be configured with IEEE 802.3ad (LACP). Internal ports are only re-activated once Spanning-tree is in Fowarding mode.

29

Configuring Trunk Fail-Over

/cfg/l2/trunk 1 ena add ext1 add ext2 /cfg/l2/failovr ON /cfg/l2/failovr/trigger 1 limit 0 ena /cfg/l2/failovr/trigger 1/amon addtrnk 1
Create a normal Trunk. (i.e.: LACP or EtherChannel)

Enable the Fail-Over feature.

Create a Trigger Profile.

Configure the number of required active link in the Trunk. 0 means that all links needs to be Down for the trigger to activate.

Apply the trigger conditions to your Trunk.

30

Team 1 BC-H
7 PS 2 3 8 PS 1 4 PS 2 3
OK OK

Team 2 BC - H
7 PS 1 8 4
OK OK

Team 3 BC
PS 2 3
OK

Team 4 BC-H
7 PS 1 8 4
OK

Link

Link

Link

TX/RX

TX/RX

TX/RX

Link

Link

Link

1
TX/RX Link

1
TX/RX Link

OK

1
OK
TX/RX Link

2
TX/RX Link

2
TX/RX Link

AC DC
Link

Link

2
TX/RX Link

3
TX/RX Link

3
TX/RX Link

TX/RX

3
TX/RX Link

4
TX/RX

4
TX/RX

4
TX/RX

IP

IP

1
TX/RX

MM 1

MM 1

Link

2
TX/RX Link

3
TX/RX Link

2
OK

MM 2
OK

MM 2

4
TX/RX

IP

2
OK

Unavailable
IP

MM 1

MM 2

OK

Link

Link

1
TX/RX Link

1
TX/RX Link

AC DC

Link

1
TX/RX Link

2
TX/RX Link

2
TX/RX Link

2
TX/RX Link

3
TX/RX Link

3
Link TX/RX Link

3
TX/RX

1
TX/RX Link

Link

4
TX/RX

4
TX/RX

4
TX/RX

2
TX/RX Link

3
TX/RX Link

4
TX/RX

10

10

10

MM: ESM1: ESM2:

192.168.70.10 192.168.70.11 192.168.70.12

MM: ESM1: ESM2:

192.168.70.20 192.168.70.21 192.168.70.22

MM: MMint: ESM1: ESM2:

192.168.70.30 192.168.70.35 192.168.70.31 192.168.70.32

MM: ESM1: ESM2:

192.168.70.40 192.168.70.41 192.168.70.42

Team 5 BC
PS 2 3
OK

Team 6 BC-H
7 PS 1 8 4
OK

Team 7 BC
PS 2 3
OK

Team 8 BC-H
7 PS 1 8 4
OK

Link

Link

TX/RX

TX/RX

Link

Link

OK

1
OK
TX/RX Link

OK

1
OK
TX/RX Link

AC DC
Link

Link

2
TX/RX Link

AC DC
Link

Link

2
TX/RX Link

TX/RX

3
TX/RX Link

TX/RX

3
TX/RX Link

4
TX/RX

4
TX/RX

IP

IP

1
TX/RX Link

1
TX/RX

2
TX/RX Link

MM 1

Link

2
TX/RX Link

MM 1

3
TX/RX Link

3
TX/RX Link

4
TX/RX

IP

2
OK

MM 2

4
TX/RX

IP

2
OK

MM 2

Unavailable
OK

OK

AC

Link

1
TX/RX Link

AC DC

Link

1
TX/RX Link

DC

2
TX/RX Link

2
TX/RX Link

3
Link TX/RX Link

3
TX/RX

Link

1 2

1
TX/RX Link

Link

TX/RX Link

4
TX/RX

4
TX/RX

2
TX/RX Link

TX/RX Link

3
TX/RX Link

3
TX/RX Link

4
TX/RX

4
TX/RX

10

10

MM: MMint ESM1: ESM2:

192.168.70.50 192.168.70.55 192.168.70.51 192.168.70.52

MM: ESM1: ESM2:

192.168.70.60 192.168.70.61 192.168.70.62

MM: MMint ESM1: ESM2:

192.168.70.70 192.168.70.75 192.168.70.71 192.168.70.72

MM: ESM1: ESM2:

192.168.70.80 192.168.70.81 192.168.70.82

USERID/PASSW0RD

Administrator/eserver1
31