Samplellddocumentv1 150427065245 Conversion Gate02

CORE INFRASTRUCTURE MIGRATION
LOW LEVEL DESIGN
CLIENT LOGO HERE
v 0.1
RE-Solution Data Ltd

Reach | Recruit | Resolve | Refine
170 Greenford Road

Harrow
Middlesex
HA1 3QX
T +44 (0) 8450 031323
DOCUMENT CONTROL
Purpose of Document
The purpose of this document is to provide A CUSTOMER with a low level design document proposing the
design for the technology refresh taking place at their Headquarters based in Gloucestershire. This
document will cover the core network and the DC environment only specifically the Nexus environment,
all other components of the network infrastructure are out of scope. This low level design will be used for
a definitive reference and a base as to how the fixed network will be developed and implemented.
The information in this document is conveyed with the assumption that engineers possess CCNP or
equivalent knowledge of network protocols and design fundamentals. It is not intended to explain every
fine detail and every configuration line for both Catalyst and Nexus devices that will be used in the design,
as its expected that engineers understand these devices and their configurations to a decent level.
Record of Modifications
Version Date Revision Name
0.1 14/02/2015 First Draft Sean Draper
0.2 Initial Peer Review
Distribution List
Name Company Contact Details Role / Responsibility
Darren Smith Resolution darren@re-solution.london Sales Account Manager
Tom Giembicki Resolution tom@re-solution.london CTO
Contributors
Name Title Responsibility

Tom Giembicki CTO Reviewer
Referenced Documents
Document Title Description

Scope Of works Document Scope Of works
Sign Off
Name Title Signature

Darren Smith Sales Director
Copyright 2015 Re-Solution. All rights reserved. Page 2 of 32

TABLE OF CONTENTS
DOCUMENT CONTROL ..........................................................................................................................................2
TABLE OF CONTENTS ............................................................................................................................................3
TABLE OF FIGURES ............................................................................................................................................... 4
TABLE OF TABLES ................................................................................................................................................ 4
RESTRICTION ON DISCLOSURE ............................................................................................................................5
INTRODUCTION .................................................................................................................................................... 6
1 SCOPE .............................................................................................................................................................7
1.1 Project Scope ..........................................................................................................................................7
1.2 Design Scope ...........................................................................................................................................7
1.3 Document Structure ...............................................................................................................................7
2 NETWORK DESIGN ....................................................................................................................................... 8
2.1 Overview................................................................................................................................................. 8
2.2 Core and Server Aggregation ................................................................................................................ 8
2.3 Server Access Layer (FlexPod) .............................................................................................................. 8
2.4 Device Abbreviations ............................................................................................................................. 9
2.5 Example Configurations ........................................................................................................................ 9
2.6 Overall Physical Diagram ...................................................................................................................... 10
3 CISCO 4500 HARDWARE .............................................................................................................................. 11
3.1 Overview................................................................................................................................................. 11
3.2 Catalyst 4500- Physical Attributes......................................................................................................... 11
3.2.1 Hardware Components .................................................................................................................. 11
3.2.2 Physical Specifications .................................................................................................................... 11
3.2.3 Physical Installation ....................................................................................................................... 12
3.2.4 Power Requirements ..................................................................................................................... 12
3.2.5 Cooing Requirements .................................................................................................................... 12
3.3 Catalyst 4500-X Software Attributes ................................................................................................... 12
3.3.1 Software Revision .......................................................................................................................... 12
3.3.2 Licensing......................................................................................................................................... 12
4 NEXUS 5000 HARDWARE ............................................................................................................................ 13
4.1 Nexus 5672UP Hardware Attributes .................................................................................................... 13
4.1.1 Hardware Components ................................................................................................................. 13
4.1.2 Physical Specifications ................................................................................................................... 13
4.1.3 Power Requirements ..................................................................................................................... 13
4.2 Nexus 5548 Software Attributes.......................................................................................................... 13
4.2.1 Software Revision .......................................................................................................................... 13
4.2.2 Licensing......................................................................................................................................... 13
5 NEW TECHNOLOGIES .................................................................................................................................. 14
5.1 Overview................................................................................................................................................ 14
5.2 Virtual Switching System (VSS) ............................................................................................................ 14
5.3 VSS Components ................................................................................................................................... 14
5.4 VSS Domain ID & Switch Numbers ....................................................................................................... 14
5.5 Virtual Switch Link (VSL) Port-Channel and Ports ............................................................................... 15
5.6 VSS Dual Active Link Detection ............................................................................................................ 15
5.7 Converting the Switch to Virtual Switch Mode ................................................................................... 15
5.7.1 VSS - Single Homed Device Considerations .................................................................................. 15
5.8 Virtual Port Channels (vPC) .................................................................................................................. 16
5.8.1 Overview ........................................................................................................................................ 16
5.8.2 vPC Components............................................................................................................................ 16

5.8.3 vPC Configuration .......................................................................................................................... 17
5.8.4 vPC Considerations ....................................................................................................................... 20
5.8.5 vPC Single Homed Devices (Orphan Ports) .................................................................................. 21
5.8.6 HSRP ............................................................................................................................................... 21
6 STANDARD PRACTICE .................................................................................................................................22
6.1 Overview................................................................................................................................................ 22
6.2 Layer 2.................................................................................................................................................... 22
6.2.1 VLAN naming and ID assignment.................................................................................................. 22
6.2.2 VLAN Configuration (Core) ........................................................................................................... 22
6.2.3 VTP .................................................................................................................................................. 22
6.2.4 CDP ................................................................................................................................................. 23
6.2.5 Spanning-Tree Protocol ................................................................................................................. 23
6.2.6 Link Aggregation........................................................................................................................... 26
6.3 Layer 3.................................................................................................................................................... 27
6.3.1 HSRP and SVI interfaces ................................................................................................................ 27
6.3.2 VLSM addressing using /31 Subnets for Point-to-Point links ....................................................... 27
6.3.3 Bidirectional Forwarding Detection (BFD) .................................................................................. 28
6.4 IP Services............................................................................................................................................. 30
6.4.1 DHCP Relay .................................................................................................................................... 30
6.5 Catalyst 4500-X Specifics ..................................................................................................................... 30
6.5.1 High Availability through SSO and NSF ....................................................................................... 30
6.6 Miscellaneous Diagrams ....................................................................................................................... 31
6.6.1 Spanning Tree & FHRP................................................................................................................... 31
TABLE OF FIGURES
Figure 1 Overall Design Diagram ...................................................................................................................... 10

Figure 2 VSS Physical vs Logical Topology....................................................................................................... 14
Figure 3 vPC....................................................................................................................................................... 16
Figure 4 - Spanning Tree & FHRP Diagram ......................................................................................................... 31
TABLE OF TABLES
Table 1 - Device Abbreviations ............................................................................................................................. 9

Table 2 Catalyst 4500-X Weight ........................................................................................................................ 11
Table 3 4500-X Power & BTU Allocation........................................................................................................... 11
Table 4 - Nexus 5672UP Weight .......................................................................................................................... 13
Table 5 - Nexus 5672UP Dimensions ................................................................................................................... 13
Table 6 - Nexus 5548 Power Requirements ....................................................................................................... 13

RESTRICTION ON DISCLOSURE
All information in this document shall not be published or disclosed wholly or in part to any third party
without Resolutions prior permission in writing, which will also be held securely. These obligations shall
not apply to information that is published or becomes known legitimately from some source other than
Resolution Data Ltd

INTRODUCTION
Overview
A Customer has engaged Resolution to assist them with their core network infrastructure, which requires
an urgent technology refresh. The existing Chassis are at full capacity, out of support and are creating
bottlenecks on the corporate network. Resolution have been given the task to design and migrate to a
new core network which will meet the growing demands of their user base and business requirements
over the next 5 years.
The Core and DC environment will undergo a technology refresh as per this document. The Access Layer,
WAN and additional network services refresh such as load balancers and firewalls are out of scope in this
design. The design will cover methods of connectivity for these services only.
Approach
Due to the risk involved, the new infrastructure will be implemented in two phases:
Phase 1:
The initial phase will look to replace the existing core Catalyst 6509 switch as this is deemed the most
critical device to the enterprise infrastructure, the new equipment being installed will be integrated with
the existing server estate environment consisting of a Cisco UCS chassis. The new installed core switches
will be deployed as a VSS pair
Phase 2:
The second phase will see the additions of two Nexus 5548-UP switches. These will be configured as a vPC
pair and will make links to the newly installed core switches and the Fabric Interconnect (FI) switches for
the server estate.

1 SCOPE
1.1 Project Scope
The project scope includes the following:
Production of a Low Level Design (This Document)
Migration Plan
Test Plan
Installation and migration of the existing Catalyst core to the new Nexus core network
1.2 Design Scope

The low level design will include the following:
Overview of the existing estate
Described methods of connectivity from the Core switches to the Nexus Switches in the FlexPod environment
Design of the Enterprise Core as well as the Nexus switches
Example of best practice configuration
Enterprise 4500 and Nexus device security
Enterprise 4500 and Nexus device management
The following are considered out of scope of this document:

Data centre physical layout
Fibre optic cabling design
Power provisioning for DC and communication rooms
UPS specification
1.3 Document Structure

The remaining structure of the low level will be divided up into a number of main sections which are titled
as follows:
Network Design Overview
Cisco Catalyst 4500 Hardware
Nexus 5000 Hardware
New Technologies
Standard Practice
Core
Server Aggregation
External Access
OSPF Design
Quality Of Service
Multicast
Security
Network Management & Monitoring

2 NETWORK DESIGN
2.1 Overview
The new network design for A CUSTOMER uses a strategic selection of Cisco Catalyst and Nexus platforms
to support IP and future converged FC/FCoE protocols for the business as well as the server infrastructure.
These platforms will provide connectivity for devices that span from the core to the server aggregation
layers within the network. The platform selection will offer CUSTOMER the scalability for the long term,
superior functionality, and high levels of resilience on purpose built infrastructure that will exhibit
extremely low convergence times around link and node failures, as well as higher throughput speeds of
10Gbps utilizing the VSS and vPC technologies.
A FlexPod will be deployed for the server estate. FlexPod components include Cisco Unified
Computing System (Cisco UCS) servers, Cisco Nexus switches, and NetApp unified storage
systems. The FlexPod architecture can scale up or out and it can be optimized for a variety of
mixed workloads, in both virtualized and non-virtualized environments.
The A CUSTOMER core network will be contained to one server room only. This will be hosting the entire
core and DC infrastructure. The network infrastructure will utilize a tiered model providing core, server
aggregation and server access.
Note: The FlexPod implementation is considered as part of the overall design; however the UCS
and Netapp unified storage components are considered out of scope.
The network hardware in the new design is as follows:
2.2 Core and Server Aggregation

Cisco Catalyst 4500-X Series Switches (VSS)
2.3 Server Access Layer (FlexPod)

Nexus N5K-5672UP Switches

Given the capabilities of the Cisco 4500 Series switches, the first main section will detail the initial setup of
the 4500-X that allows the remainder of the design for A CUSTOMER to be implemented. The VSS section
is a primer for the new core aggregation technologies that are leveraged heavily in the design which
should be understood from a concept and configuration standpoint before moving further through the
document. Common configurations are grouped to save duplication, and explain the best practice
configurations for Layer 2 and Layer 3 protocols which should be kept standard across all areas
throughout the design.
The next sections will detail each sites physical and logical design for that particular network layer. This
includes hostnames, IP addressing, device types etc.
2.4 Device Abbreviations

Throughout this document there are abbreviations used for different layers or devices within the design.
As a summary these abbreviations are as follows:
Abbreviation Description
CAL Core Aggregation Layer

SAL Server Aggregation Layer
Table 1 - Device Abbreviations
2.5 Example Configurations

Example configurations throughout the remainder of this document will use both NX-OS and IOS syntax.
Not every instance will show both examples as its expected that engineers know the configuration syntax
differences between the two operating systems.

2.6 Overall Physical Diagram
The diagrams below depict the new infrastructure the project will deliver as part of the core network refresh:
Mgmt
Legend FCoE Only
Converged
10 GbE Only
Catalyst 4500X Series VSS Catalyst 4500X Series
OIR OIR
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 STATUS 1 2 3 4 5 6 7 8 OIR 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 STATUS 1 2 3 4 5 6 7 8 OIR
UID STATUS PS1 PS2 FAN USB UID STATUS PS1 PS2 FAN USB
vPC vPC
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
PO
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 1 2 3 4 9 10 11 12 17 18 19 20
N5K-C56-72UP
N5K-C56-72UP
ID
ID
5 6 7 8 13 14 15 16 21 22 23 24
STAT
5 6 7 8 13 14 15 16 21 22 23 24
STAT
PO PO
vPC vPC
L1 MGMTO FAN FAN
PS1
FAN1 L1 MGMTO FAN FAN
PS1
STAT STAT FAN1
STAT STAT
FAN2 FAN2
FAIL FAIL
FAIL FAIL
OK OK
ID ID OK OK
PS2
PS2
L2 CONSOLE
STAT STAT L2 CONSOLE
UCS B200 M3 UCS B200 M3
UCS 5108
!
SLOT SLOT
2x 10 GbE with 2x 10 GbE with

1 2
! Console Reset ! Console Reset
FCoE per FI SLOT

3
SLOT
4 FCoE per FI
Drive Slot Cover

PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
CONSOLE
!
SLOT SLOT
5 6
UCS
C240 M3
SLOT SLOT
7 8
OK FAIL OK FAIL OK FAIL OK FAIL
UCS B200 M3 UCS B200 M3
UCS 5108
2x 10 GbE with
!
SLOT SLOT
2x 10 GbE with
1 2
! Console Reset ! Console Reset
FCoE per FI SLOT

3
SLOT
4
FCoE per FI
Drive Slot Cover

PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
PWR
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
SYS
CONSOLE
!
SLOT SLOT
5 6
UCS
C240 M3
SLOT SLOT
7 8
OK FAIL OK FAIL OK FAIL OK FAIL
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
DS2246
2x 10 GbE 2x 10 GbE
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
FCoE 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
DS2246
FCoE
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
900GB
FAS8020
A B
600GB 600GB 600GB 600GB
DS4243
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
DS4243
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
600GB 600GB 600GB 600GB
Figure 1 Overall Design Diagram

3 Cisco 4500 Hardware
3.1 Overview
The Cisco Catalyst 4500-X switch is a fixed aggregation switching platform that is capable of virtualization
and integrated network services for video, security and application visibility. With support for VSS the
platform provides a single point of management whilst offering up to 1.6Tbps switching capacity.
Using a VSS deployment enables cross-chassis high-availability capabilities such as In-Service Software
Upgrade (ISSU), Nonstop Forwarding (NSF), and Stateful Switchover (SSO). Within the new design the
two Catalyst 4500-X will provide the core infrastructure utilizing a VSS design.
Note: For VSS capability, the minimum software requirement is Cisco IOS XE Software Release 3.4.0SG
3.2 Catalyst 4500- Physical Attributes

The Catalyst 4500-X comes in a one-rack unit (1RU) which lends to a low-power form factor deployment.
For A CUSTOMER Deployment, each Catalyst 4500-X will consist of the following components:
3.2.1 Hardware Components

1 x Catalyst 4500-X Chassis No PSU (WS-C4500-X-40X-ES)
2 x 750E AC Power Support (C4KX-PWR-750AC-R)
1 x Catalyst 4500-X 8 Port 10G Network Module (C4KX-NM-8SFP+)
N+1 full power redundancy will be enabled for each device
3.2.2 Physical Specifications

The following physical specifications apply to each Catalyst 4500-X switches, below is the weight of each
switch and its installed components along with dimensions of each chassis:
Height Width Depth Weight

Model Additional Parts
(U) (cm) (cm) (Kg)
WS-C4500-X-40X-ES N/A 1 43.1 53.34 8.23

N/A C4KX-NM-8SFP+ N/A N/A N/A 0.84
N/A C4KX-PWR-750AC-R N/A N/A N/A 1.68
N/A C4KX-PWR-750AC-R N/A N/A N/A 1.68
Total 53.34 12.43
Table 2 Catalyst 4500-X Weight
Output Power Switch Power Consumption Heat Dissipation

Model
(W) (W) (BTU(W)/Hr)
WS-C4500-X-40X-ES 1500 330 1122 (329 W)
Table 3 4500-X Power & BTU Allocation

3.2.3 Physical Installation
The Catalyst 4500-X should ideally be installed in a standard four post 1200mm rack so that it can
accommodate cable overhang while ensuring that the cabinet doors can be closed
The airflow of the Catalyst 4500-X switches is front to back for the power supplies, as such special
considerations need to be taken into account regarding the clearance on the front and back for the rack of
the chassis, ensuring that all slots and openings on a chassis remain unobstructed, especially the fan
assembly vent at the back of the chassis. To maintain proper air circulation through the switch chassis, it is
recommended that you maintain a minimum 6-inch (15 cm) separation between a wall and the chassis hot
air exhaust
3.2.4 Power Requirements

The power supplies for each Catalyst 4500-X are rated at 750W. Two power supplies will be used for each
chassis for full redundancy. These power supplies will be configured to operate in full-redundancy mode.
Each active PSU requires two power feeds that should be rated not greater than 20A. Feed receptacles
should be within 3.0m of each power supply.
Power Recommendations
Each power supply should be connected to separate input sources; otherwise the chassis could be
susceptible to a power failure
3.2.5 Cooing Requirements

A single Catalyst 4500-X chassis in the data centre will dissipate approximately 1122 BTUs per hour. Its
important that each chassis is placed in the appropriate position in racks/isles for optimal cooling. This will
ensure that the system fans do not need to work excessively to cool the system. The air flow design of the
4500-X chassis is front-to-back for the power which suits hot isle cold isle data centres.
3.3 Catalyst 4500-X Software Attributes

The following sections describe the software version and licensing that will be used on each chassis.
3.3.1 Software Revision

The software revision to be used on the Nexus 7000 is version IOS XE 3.4.4SG and this is the
recommended version for now. This will also allow support for VSS which is required as per the design
3.3.2 Licensing
Each Catalyst 4500-X will chassis have the Enterprise Services which will allow for every feature
available in the 4500-X platform

4 NEXUS 5000 HARDWARE
4.1 Nexus 5672UP Hardware Attributes
The following sections describe the hardware specification for each Nexus 5600 switch.
4.1.1 Hardware Components

1 x Nexus 5672UP Switches
2 x 1100W Power supplies
N+1 full power redundancy will be enabled for all switches.
4.1.2 Physical Specifications

The following physical specifications apply to each of the Nexus 5672UP switch which will have an impact
on the installation of each device. Shown below is the weight of each switch.
Quantity Part Weight

N5K
1 N5K-C5672UP 14.51Kg
Total 14.51Kg
Table 4 - Nexus 5672UP Weight
Displayed below are the physical dimensions of each Nexus 5672UP chassis:
Height Width Depth

N7K
4.4cm 43.9cm 74.9cm
Table 5 - Nexus 5672UP Dimensions
4.1.3 Power Requirements

The table below lists the power requirement for each of the Nexus 5672UP chassis that will be installed:
Input Power Supply Typical Power Maximum Power Heat

Ports Model Current Rating Used Used Dissipation
(A) (W) (W) (W) (BTU/Hr)
N5K-
48 3.75 750 390 600 1331
C5672UP
Table 6 - Nexus 5548 Power Requirements
4.2 Nexus 5548 Software Attributes

The following sections describe the software version and licensing that will be used on each chassis.
4.2.1 Software Revision

The software revision to be used on the Nexus 5672UP is version 6.0(2)N1(2a) and this is the
recommended version for now and is also in use at A CUSTOMER 72 Hammersmith Rd site.
4.2.2 Licensing
No special licensing is required for the Nexus 5000 series switches as these switches are acting as pure
Layer 2 switches. The switches will be running the standard LAN Base feature set.

5 NEW TECHNOLOGIES
5.1 Overview
Given that some new technologies are being introduced into the A CUSTOMER network to support the
new design, this section aims to cover new concepts and features that are later referenced in the site
specific sections under layer 2 and layer 3 designs. These features are applicable to key components within
each topology and its highly advisable that readers become familiar with this section prior to reading the
remaining parts of this document.
Each technology shown in the following sections includes configuration examples that are based on Cisco
best practices which should be adhered to throughout the network design unless otherwise stated. This
information will not be mentioned again so it should be known that where remaining parts of the
document mentions a technology (which is mentioned in this section), the configuration will be more or
less the same.
5.2 Virtual Switching System (VSS)
A VSS combines a pair of Catalyst 4500 or 4500-X series switches into a single network element. The VSS
manages the redundant links, which externally act as a single port channel.
The VSS simplifies network configuration and operation by reducing the number of Layer 3 routing
neighbours and by providing a loop-free Layer 2 topology.
Figure 2 VSS Physical vs Logical Topology

5.3 VSS Components
5.4 VSS Domain ID & Switch Numbers
The VSS domain must match between both switches. The virtual switch domain is a number between 1
and 255, and must be unique for each VSS in your network (the domain number is incorporated into
various identifiers to ensure that these identifiers are unique across the network).
Within the VSS, you must configure one switch to be switch number 1 and the other switch to be switch
number 2. To configure the virtual switch domain and switch number on both switches, perform these
tasks:
Config Example
Switch-1(config)# switch virtual domain 100

Switch-1(config-vs-domain)# switch 1

Switch-2(config-vs-domain)# switch 2

5.5 Virtual Switch Link (VSL) Port-Channel and Ports
The VSL link is formed between the two chassis with the VSS domain. This link is forms a port-channel that
is used to pass data and control traffic between the two switches within the domain. To configure the
virtual switch link on both switches, perform these tasks:
.
Config Example
Switch-1(config-interface)# interface range tengigabitethernet 4/1-2
Switch-1(config-if)# channel-group 10 mode on
Switch-1(config)# interface port-channel 20
Switch-1(config-interface)# switch virtual link 1
Switch-2(config-interface)# interface range tengigabitethernet 4/1-2

Switch-2(config-if)# channel-group 10 mode on
Switch-2(config)# interface port-channel 20
Switch-2(config-interface)# switch virtual link 2
5.6 VSS Dual Active Link Detection

The VSS switches require a dual active detection mechanism to protect against a split brain scenario in the
event of a VSL link failure. Fast Hellos will be configured across the VSS pair to support the dual active
failure protection. To configure Dual Active Link Detection on both switches, perform these tasks:
Config Example

Switch-1(config-vs-domain)# dual-active detection fast-hello
Switch-1(config)# interface gigabitethernet 1/5/1
Switch-1(config-if)# dual-active fast-hello

Switch-2(config-vs-domain)# dual-active detection fast-hello
Switch-2(config)# interface gigabitethernet 1/6/1
Switch-2(config-if)# dual-active fast-hello
5.7 Converting the Switch to Virtual Switch Mode

Once all of the above has been configured you can convert the single chassis into a VSS. To convert from a
single switch to a VSS on both switches, perform these tasks:
Config Example
Switch-1# switch convert mode virtual

Switch-2# switch convert mode virtual
Note After you confirm the command (by entering yes at the prompt), the running configuration is
automatically saved as the startup configuration and the switch reboots. After the reboot, the switch is in
virtual switch mode, so you must specify interfaces with three identifiers (switch/module/port). When
switches are being converted to VSS, you should not set them to ignore startup-config. If done, the switch
can be enabled to parse the startup-config at the rommon prompt. Ignoring startup-config in VSS mode,
causes a switch to boot in a semi-VSS mode, which can only be corrected by a reboot and by enabling the
parsing of startup-config.
5.7.1 VSS - Single Homed Device Considerations

Devices that are single homed into a single VSS member switch are known as Orphan Ports. In the event
of a VSL link failure, the VSS system will shut down all ports connected to the operation primary switch. If
the switch that is deemed primary has orphan port connections, then these ports will be shutdown
causing a loss of service to these devices.
The VSL link failure is a rare scenario, but should be protected against and all devices should be dual
homed across each of the VSS member switches. The desired design is to have no orphan ports connected
to the VSS domain.

5.8 Virtual Port Channels (vPC)
5.8.1 Overview
Virtual PortChannels (vPCs) allow links that are physically connected to two different Cisco switches to
appear to a third downstream device as coming from a single device and as part of a single port channel.
The downstream device can be a switch, a server, or any other networking device that supports IEEE
802.3ad Port Channels.
vPC allows the creation of Layer 2 PortChannels that span two switches and at present vPC is
implemented on the Nexus 7000 and 5000 series platforms running NX-OS version 4.1(4) or higher.
Benefits of vPC are device level redundancy with faster convergence than using spanning tree, elimination
of blocked ports which promotes a loop-free topology, and much better bandwidth utilisation.
5.8.2 vPC Components

vPCs consist of two vPC peer switches, which are referred to as Switch 1 and Switch 2 in the diagram
below. Of the vPC peers one is primary and the other is secondary. The system formed by Switch 1 and
Switch 2 is referred to as a vPC domain.
Figure 3 vPC
The vPC peer switches are connected through a link called peer link, also known as multi-chassis
PortChannel trunk. This is a standard 802.1Q trunk that carries vPC and non vPC VLANs, Cisco Fabric
Services messages (which are tagged as CoS 4), flooded traffic from the other vPC peer, and other
protocols like STP BPDUs, IGMP joins/updates, HSRP hellos etc.
The ports that form the PortChannel are split between the vPC peers and are referred to as vPC member
ports.
An out-of-band link is used to resolve dual-active scenarios where the peer-link connectivity is lost. This
link is referred to as vPC peer-keepalive or fault-tolerant link.
Some devices can be single-attached to the topology, like server 1 and server 3. The ports connecting
devices in a non-vPC mode to a vPC topology are called orphaned ports. Design considerations often
surround these types of ports as its extremely important for the design.

5.8.3 vPC Configuration
The following sections show how to correctly configure a vPC domain and attach downstream devices.
vPC Role and Priority

First step in configuring vPC is to enable the feature vpc and then a domain needs to be defined (as
indicated by a domain-id) as well as priorities to define primary and secondary roles in the vPC
configuration. The lower number in a vPC priority is more preferred Generally this is should be aligned
with the spanning-tree priorities in the topology for ease of management.
While mismatched Spanning-Tree and vPC priorities do not any impact on traffic forwarding under normal
conditions, it is desirable to keep the priorities matched so as to have Spanning-Tree root and vPC primary
on the same device and Spanning-Tree secondary root and vPC secondary on the same device.
For the two switches (vPC peers) to form a vPC system, the domain-id of these switches need to match.
The domain-id is used to generate the system-id which is used for the LAGID in the LACP negotiation and
BPDU generation.
Shown below is the configuration to create and set the vPC domain, and set the matching STP priorities:
N5K-1(config)# feature vpc

N5K-1(config)# vpc domain 1
N5K-1(config-vpc-domain)# role priority 4096
N5K-2(config)# feature vpc

N5K-2(config-vpc-domain)# role priority 8192
It should also be noted that the vPC role is not preemptive, so a device may be operationally primary but
secondary from a configuration perspective. This is not an issue.
vPC Peer Keepalive

Next step is to configure the mechanism for dual-active detection. The keepalive that resolves dual-active
scenarios should never be carried as a VLAN on the peer link. It should be carried over separate routed
infrastructure. The reason for this is if the peer-link fails and the keepalive fails at the same time, you will
have split brain and a broken network topology.
The recommendation is to use the Nexus out-of-band management interface for this task. However, you
should not use the mgmt0 interface for a direct back-to-back connection between Nexus 7000 systems
because the mgmt0 IP address is only ever active on one supervisor, and you cannot discern which
supervisor is active at any given time. For the Nexus 5600 this is acceptable as only one mgmt interface
exists, but not recommended as it means device management would have to be in-band.
The keepalive itself is a UDP message on port 3200, is 96 bytes long (32 byte payload), and includes
version, time stamp, local and remote IPs, and the domain ID.
The following configuration shows how to use the management interface for peer-keepalive. This
assumes that out-of-band management hasnt currently been set up:
N5K-1(config)# vrf context management

N5K-1(config-vrf)# ip route 0.0.0.0/0 192.168.0.1
N5K-1(config)# interface mgmt0

N5K-1(config-if)# ip address 192.168.0.2/24
N5K-1(config-if)# no shutdown
N5K-1(config-if)# exit

N5K-1(config-vpc-domain)# peer-keepalive destination 192.168.0.3 source 192.168.0.2 vrf management

vPC Peer-Link
A PortChannel is used to connect the two vPC devices together similar to a normal ISL, but instead its
referred to as the peer-link. It still carries VLANs just like an ISL would. The peer-link link also carries
additional traffic such as BPDUs and HSRP hellos, and MAC address synchronization between the vPC
peers using CFS (Cisco Fabric Services).
On the N7K Series, this PortChannel should be configured on 10 Gigabit Ethernet interfaces configured in
dedicated-mode across two different 10 Gigabit Ethernet line cards. However there are exceptions to this
which will be explained in later site specific sections.
The peer-link is by far the most important component in the vPC system, in that its failure may impair the
establishment of new flows and isolate orphan ports.
Configuring the peer link in a redundant fashion ensures uninterrupted connectivity between the vPC
peers in almost every likely failure scenario.
The following configuration illustrates how to configure the peer-link across two 10Gbps interfaces which
in this case is PortChannel1 using LACP active negotiation at both ends.
N5K-1(config)# feature lacp
N5K-1(config)# interface Port-Channel1

N5K-1(config-if)# switchport
N5K-1(config)# interface E1/1 , E2/1

N5K-1(config-if)# channel-group 1 mode active
N5K-1(config-if)# no shut
N5K-1(config)# interface Port-Channel1

N5K-1(config-if)# switchport mode trunk
N5K-1(config-if)# switchport trunk allowed vlan <vlans>
N5K-1(config-if)# vpc peer-link
Note: You should always create a Port-Channel interface first and specify if its Layer 2 or
Layer 3 using the switchport or no switchport command before bundling any links.
vPC Member Ports

PortChannels are configured by bundling ports on each Nexus switch through the command vpc, as
shown in the following configuration example. The system will issue an error message if the PortChannel
wasnt previously configured as a switchport as its a requirement of vPC to operate only with Layer 2
interfaces.
It is always good practice to use the Link Aggregation Control Protocol (LACP) for bundling of the ports in
the vPC. This is because LACP determines that the ports being bundled are actually part of the same
physical or virtual switch, preventing erroneous configurations.
The configuration shown below places a single 10Gbps interface on each vPC peer into a PortChannel and
then applies vPC to the PortChannel on each peer. Following this is an example configuration of a
PortChannel on a downstream device such as a Catalyst 2950:
N5K-1(config)# interface Port-channel10

N5K-1(config)# interface Ethernet1/7


N5K-1(config-if)# vpc 10


N5K-2(config)# interface Ethernet1/7


N5K-2(config-if)# vpc 10
Downstream device configuration:

C2950(config)# interface range GigabitEthernet1/0/49 , GigabitEthernet2/0/49
C2950(config-if)# switchport
C2950(config-if)# channel-group 10 mode passive
C2950(config)# interface Port-channel 10

C2950(config)# no shut
Note: The vpc ID doesnt need to be the same as the PortChannel ID, however its recommended; also the
recommended configuration of downstream devices is to use passive negotiation.
vPC Consistency Checks

Consistency checks refer to a number of configuration parameters on the PortChannels (both member
and peer-links) that must be identical on both peers in the domain. If any of these values are mismatched
it can cause the vPC formation to fail. Examples of these are allowed VLANs, speed, duplex, spanning-tree
port types, and port mode (switchport / trunk).
To view these you can use the show vpc consistency-parameters command set.
vPC Reload Restore

In major outage scenarios, both Nexus 7000 devices that make up a vPC domain could be reloaded. During
this scenario, occasionally only one of the peers can be restored. This is because the vPC port channels
cannot come up if no handshake has ever been performed with its remote peer.
A new feature in NX-OS 5.0(2) allows you to configure the Nexus 7000 to restore vPC services when its
remote peer fails to come online.
Its recommended to use the command reload restore on both vPC peers to avoid this rare scenario.
vPC Peer Gateway

In NX-OS you can configure vPC peer devices to act as the gateway for packets that are destined to the
vPC peer device's MAC address.
Particularly with some network-attached storage (NAS) devices or load-balancers, they may have features
which are aimed at optimising application performance. Behaviour of these devices will reply to traffic
using the MAC address of the sender Cisco Nexus 7000 device rather than the common HSRP gateway.
This behaviour is non-complaint with some basic Ethernet RFC standards. Packets reaching a vPC device
for the non-local router MAC address are sent across the peer-link and could be dropped by the built in
vPC loop avoidance mechanism if the final destination is behind another vPC.
The vPC peer-gateway capability allows a vPC switch to act as the active gateway for packets that are
addressed to the router MAC address of the vPC peer. This feature enables local forwarding of such
packets without the need to cross the vPC peer-link. In this scenario, the feature optimizes use of the
peer-link and avoids potential traffic loss.

Configuring the peer-gateway feature needs to be done on both primary and secondary vPC peers and is
non-disruptive to the operations of the device or to the vPC traffic. The vPC peer-gateway feature can be
configured globally under the vPC domain.
The configuration is shown below:

N5K-1(config)# peer-gateway
Note: When enabling this feature it is also required to disable IP redirects on all interface VLANs mapped
over a vPC VLAN to avoid generation of IP redirect messages.
vPC Peer-switch
The vPC peer switch feature addresses performance concerns around STP convergence. This feature
allows a pair of Cisco Nexus devices to appear as a single STP root in the Layer 2 topology and it eliminates
the need to pin the STP root to the vPC primary switch as well as improving vPC convergence if the vPC
primary switch fails.
To avoid loops the vPC peer link is excluded from the STP computation. In vPC peer switch mode, STP
BPDUs are sent from both vPC peer devices to avoid issues related to STP BPDU timeout on the
downstream switches, which can cause traffic disruption.
Below is a configuration example:
NX-OS
N5K(config)#vpc domain 1
N5K(config-vpc-domain)#peer-switch
Note: RSTP VLANs priority on both switches must be the same.
5.8.4 vPC Considerations
Attaching to a vPC domain

When attaching devices to a vPC domain you should always dual-home them to both vPC peers. There are
cases where this cannot happen such as using load balancers or redundant firewalls, and if so appropriate
measures should be taken to ensure that during certain failure scenarios that single homed devices are
unaffected, and traffic flow is not black holed.
Double Sided vPC

Double sided vPC is the attachment of a pair vPC devices to one another pair and utilising the technology
to create a single logical link between them. An example of this is a pair of Nexus 5600 switches attached
to a pair of Nexus 7000, with a full mesh of links aggregated into a single PortChannel.
As mentioned earlier domain-id defined for the vPC domain is used to generate the system-id of the
system comprised of the vPC peers. Because of this it is important to ensure that each vPC system in a
topology such as the one described utilizes a different domain-id number. This ensures uniqueness in the
Bridge ID used for BPDUs and the LAGID used by LACP. Alternatively its possible to define the same
domain-id as long as a different system-id (system-priority) has been configured under the vPC domain.
Its recommended to configure the system-priority manually the same as the domain ID.

5.8.5 vPC Single Homed Devices (Orphan Ports)
Overview
In a vPC environment the most important link in the technology is the Peer-Link. If the Peer-Link fails for
whatever reason you have the potential for a split brain scenario where both vPC peers think everything is
normal, and continue to advertise subnets to non-vpc peers and continue forwarding on downlinks (vPC
member ports). This behavior is avoided by the use of the Peer Keepalive which determines if both peers
are still responsive on the out-of-band link, and if so, the currently acting secondary role will shut all of its
vPC member ports, and bring down all of its SVI interfaces for any VLANs that are configured across the
Peer-Link. This behavior is fine for situations when all devices are dual homed to the vPC domain and Port-
Channeled to both peers.
If you have single homed devices attached to the domain, you need to think carefully about this failure
scenario and determine if the failure of the peer-link would cause extended loss of service because its
broken the path of traffic, and cannot be resolved without manual intervention.
Within this design vPC is only configured on the server aggregation layer of the network, where devices
are dual homed.
When we consider the possibility of losing the vPC Peer-Link either by misconfiguration, or disconnected
fibers etc. The secondary vPC peer will shut its member ports because the keep alive has identified that
both are still reachable. For A CUSTOMER network devices are dual homed into both the core and server
aggregation Nexus 7000 switches.
5.8.6 HSRP
The use of HSRP in the context of vPC does not require any special configuration. With vPC, only the active
HSRP interface answers ARP requests, but both HSRP interfaces (active and standby) can forward traffic.
If an ARP request coming from a server arrives on the secondary HSRP device, it is forwarded to the active
HSRP device through the peer-link.

6 STANDARD PRACTICE
6.1 Overview
The following section goes through standard practices which should remain consistent across the design
and on-going unless there is a good reason to change them. This is broken down into Physical, Layer 2, and
Layer 3 technologies which are typically Layer 2 and Layer 3 protocols such as spanning-tree, HSRP etc.
This section is mainly to avoid duplication of information throughout every section.
If configurations are pertinent to certain network layers it will be specified otherwise the section is global
to each network layer and/or site. Reason for this is that different areas of the network require different
policy and security or trust levels.
Devices covered will include:

NX-OS: Nexus 5600
IOS: Catalyst 4500
6.2 Layer 2
6.2.1 VLAN naming and ID assignment

When creating new VLANs they should have a naming convention to easily identify their function, and the
actual VLAN ID should ideally be unique to the site. Exceptions to this are when Layer 2 extension is
required between two locations using technologies such as OTV. OTV is not used within this phase of the
design and may be looked at in future. Existing VLAN names will be used, unless absolutely required no
additional VLANs will be created.
6.2.2 VLAN Configuration (Core)

A CUSTOMER have VLANs that span to many of the communication rooms scattered around the campus.
There is no requirement to redesign the VLAN scheme and is out of scope, as such the VLAN design for A
CUSTOMER will not change for the core/access layer and server aggregation/server access data networks,
it is important to mention that these 2 network segments will be separated by the use of VDCs and
connectivity between the two will be achieve with Layer 3 links.
6.2.3 VTP
VTP is a Layer 2 messaging protocol that maintains VLAN configuration consistency by managing the
addition, deletion, and renaming of VLANs within a VTP domain. Nexus switches will support the standard
VTP modes of client, server, and transparent, with an additional mode of vtp off. The main difference
with VTP being turned off is that the device will not relay any VTP messages between devices. A
CUSTOMER makes use of VTP to automatically configure VLANs on switches within the same VTP domain,
this function ability will be enabled on the nexus switches which by default have it turned off. Within the
design it is recommended that the core/access layer and the server aggregation/server access layer reside
in separate VTP domains to prevent accident deletion of VLANs should Layer 2 links be added between the
two domains. The Nexus 7000 switches will act as the VTP servers and all VLANs will be created in the
appropriate VDCs, all other switches in the network will act as VTP clients.
The VTP Server configuration on Nexus:

N5K-1(config)# feature vtp
N5K-1(config)# vtp mode server
N5K-1(config)# vtp domain Domain01
N5K-1(config)# vtp version 2
N5K-1(config)# vtp password Password123

6.2.4 CDP
CDP is a device discovery protocol that runs over Layer 2 (the data link layer) on all Cisco-manufactured
devices (routers, bridges, access servers, and switches) and allows network management applications to
discover Cisco devices that are neighbours of other already-known devices. CDP should be left enabled
when facing TRUSTED devices as it assist with troubleshooting and is required for the implementation of
Wireless APs and IP phones.
6.2.5 Spanning-Tree Protocol

Spanning Tree Protocol provides a mechanism for physically redundant network topologies to remain
loop free. All devices in a Layer 2 domain run the spanning-tree calculations to discover the topology and
calculate the best path to the root bridge. Through the spanning-tree process redundant network links are
placed into a blocking state preventing loops from occurring at Layer 2.
The recommended spanning-tree protocol across the board for A CUSTOMER is Rapid-PVST protocol.
Root Bridge Placement & STP Load Balancing (Core / Server Aggregation)
For every instance of spanning-tree (per-VLAN) there is one device elected Root Bridge. The root bridge is
typically at the centre of the network topology and all of its Layer 2 links in forwarding state.
In a typical network design, spanning-tree is also used to provide load balancing across Layer 2 paths
where multiple VLANs exist. This normally achieved by configuring multiple devices to act as the root
bridge, and then mapping the root bridge with other protocols such as HSRP.
Root bridge placement in vPC deployments is a much more simplified configuration because one of the
key characteristics of vPC is active/active forwarding on Layer 2 paths. In this scenario the root bridge
placement is less of a concern since load balancing is an inherent part of vPC and the use of the peer-
switch command.
As such the root bridge placement in every Layer 2 domain of the A CUSTOMER design is to use the same
priority for all VLANs to Nexus 7000 VDCs, and make the root priorities the same as the vPC domain
priorities where appropriate.
Config Example:
vPC Primary
N5K-1(config)# spanning-tree mode rapid-pvst
N5K-1(config)# spanning-tree vlan 1-4096 priority 4096
vPC Secondary
N5K-2(config)# spanning-tree mode rapid-pvst
N5K-2(config)# spanning-tree vlan 1-4096 priority 4096
Spanning Tree behaviour with vPC

There are two key behavioural differences with spanning tree in a vPC environment:
vPC imposes the rule that the peer link should never be blocking because this link carries important traffic such as the Cisco
Fabric Services over Ethernet (CFSoE) Protocol. The peer link is always forwarding.
For vPC ports only, the operational primary switch generates and processes BPDUs. The operational secondary switch forwards
BPDUs to the primary switch.

Spanning Tree Path Cost
In a regular spanning-tree environment with no additional configurations, the cost of most links in a data
centre is almost identical regardless of the available bandwidth along a given path:
1Gbps link = 4
2Gbps links = 3
3Gbps links = 3
4Gbps links = 3
Considering that spanning tree was designed before the availability of 10 GigabitEthernet or even
GigabitEthernet, the default spanning-tree reference cost for a link is inadequate, as most links with 10
GigabitEthernet bandwidth or bandwidth with multiples of 10 Gigabit Ethernet will appear to have the
same cost.
With spanning-tree pathcost method long enabled, spanning tree calculates the best forwarding path
according to the link bandwidth to the root and not based on hop count. With this command, the cost of
links is as follows (these numbers are weights so they dont have a specific measurement unit):
One Gigabit Ethernet link = 20,000
Two Gigabit Ethernet links = 10,000
Three Gigabit Ethernet links = 6660
Four Gigabit Ethernet links = 5000
The cost with 10 Gigabit Ethernet links is as follows:

One 10 Gigabit Ethernet link = 2,000
Two 10 Gigabit Ethernet links = 1,000
Path cost method long should be enabled on every device participating in the spanning tree. This will be
enabled for all devices being installed as part of the core LAN refresh, however all remaining switches such
as the access layer switches remain the responsibility of the A CUSTOMER IT team to update.
Spanning-Tree Portfast
Portfast immediately brings an interface configured as an access or trunk port to the forwarding state
from a blocking state, bypassing the listening and learning states. Portfast should be used on interfaces
connected to a single workstation or server, to allow those devices to immediately connect to the
network, rather than waiting for the spanning tree to converge (up to 50 seconds delay).
NX-OS(config)# interface ethernet 2/1

NX-OS(config-if)# spanning-tree port type edge
OR
IOS(config)# interface GigabitEthernet 0/1

IOS(config-if)# spanning-tree portfast

NX-OS Spanning-Tree Port Types (Core /Aggregation)
In order to ease the administration of spanning-tree there are a number of port types that can be
associated with network port based on their function in the design.
Normal ports - By default a switchport is considered a normal port for the purpose of spanning-tree. Normal ports remain
unmodified and operate as standard spanning-tree ports
Network ports - Network ports are used to define connections between two bridges. By default Bridge Assurance is enabled on
these ports, which is described in the next section.
Edge ports - Previously known as Portfast, a port configured as a spanning-tree edge should transition immediately into a
forwarding state, bypassing the listening and learning states. Only non-bridging L2 devices should be configured as edge ports.
This port type should be reserved for data centre hosts which are not capable of creating a L2 loop; this includes single
attached hosts, L3 routers and firewalls, or multi-homed devices that leverage some form of NIC teaming. Trunk ports to these
types of devices can also be configure as edge ports (this is called TrunkFast and generally used for virtualisation hosts).
It is recommended that all ports are left in the default configuration of normal so that spanning-tree will
protect the topology when needed, and explicit configuration is given for any ports that should be
network or edge.
To configure port types the following configuration is used:
N7K(config)# interface ethernet 2/11

N7K(config-if)# spanning-tree port type network
OR

N7K(config-if)# spanning-tree port type edge
Warning: Edge port type (portfast) should only be enabled on ports connected to
a single host. Connecting hubs, concentrators, switches, bridges, etc...
to this interface when edge port type (portfast) is enabled, can cause temporary
bridging loops.
Use with CAUTION
OR

N5K(config-if)# spanning-tree port type edge trunk
BPDU Guard
BPDU guard prevents a host port (port type edge or portfast) from participating in spanning tree. Under
normal circumstances, Layer 2 access ports connected to a single workstation or server should not
participate in spanning tree. When enabled on a port, BPDU guard shuts down the port as soon as a BPDU
is received on that port. In this way, BPDU guard helps prevent unauthorized access and the illegal
injection of forged BPDUs.
BPDU guard can be configured per port, or globally. When configured globally, BPDU guard is effective
only on ports in the operational port type edge mode. Its recommended that global BPDU Guard be
enabled across the A CUSTOMER infrastructure therefore affecting all ports that are placed in to edge
mode.
Global BPDU Guard for edge or Portfast ports is enabled as follows:
NX-OS(config)# spanning-tree port type edge bpduguard default
IOS(config)# spanning-tree portfast edge bpduguard default

IOS(config)# spanning-tree portfast bpduguard default
BPDU guard should also be enabled on individual interfaces that are in portfast or edge mode:
NX-OS(config)# interface E100/1/1

NX-OS(config-if)# spanning-tree bpduguard
STP Root Guard

STP Root Guard enforces the placement of the root bridge. STP root guard is a feature that is enabled on
selected ports to prevent surrounding switches (usually downstream) from becoming the root switch.
The root guard feature forces a port to become a designated port so that no switch on the other end of
the link can become a root switch. If a port configured for root guard receives a superior BPDU, the port
immediately goes into a root-inconsistent (blocked) state. In this way, STP root guard blocks other
devices trying to become the root bridge by sending superior BPDUs.
STP Root guard should be enabled on all downlinks away from the root bridge in each network layer,
server access ports, and user access ports.
Root guard can be enabled with the following command:
switch(config)# interface ethernet 2/1

switch(config-if)# spanning-tree guard root
STP Loop Guard

STP Loop guard prevents alternate or root ports from becoming the designated port due to a failure that
could lead to a unidirectional link. Loop guard operates only on ports that are considered point to point
by spanning tree.
Root guard is mutually exclusive with Loop guard, as Root guard is to be used on designated ports and
does not allow the port to become non-designated. Loop guard works on non-designated ports and does
not allow the port to become designated via max_age expiry.
Its recommend that Loop guard be enabled on root and alternate ports for all possible combinations of
active topologies where Bridge Assurance is not supported. An example of this would be the 2960
switches.
Configuration of loop guard shown below:
switch(config)# interface GigabitEthernet1/0/1

switch(config-if)# spanning-tree guard loop
6.2.6 Link Aggregation

Where the aggregation of links is a requirement, PortChannel should be used to create a logical bundle.
LACP should be used for the PortChannel management. A total of 8 physical ports maybe used as part of
any single PortChannel configuration.
LACP has the same characteristics as PAgP, which is Cisco proprietary. As PAgP and LACP are not
interoperable, LACP will be the preferred negotiation protocol within A CUSTOMER.
LACP will be set to active/passive mode and the medium type specified as point-to-point. The active device
should always be closer towards the Core of the network with the other end as passive.
Config Example:
switch(conf-if)# channel-group 1 mode active

switch(conf-if)# channel-protocol lacp
Note: PortChannel active/active or active/passive will be preferred where it is supported
When bundling links its important to ensure that they are all the same speed, their duplexing mode is full,
and any configurations are mirrored across all ports. Additionally, PortChannel configurations should
always be consist ports to the power of 2 to provide the best load distribution.
6.3 Layer 3
6.3.1 HSRP and SVI interfaces

Hot-standby Routing Protocol (HSRP) provides high network availability and routes IP traffic from hosts
on a particular LAN segment without relying on the availability of any single router. HSRP is configured
across a number of routers (or routed interfaces) where the selection of an active and standby router
takes place. The active and standby router is selected based upon the HSRP priority set on each device.
When configuring priorities the best practice is to align the active HSRP device to the primary root bridge
or vPC primary switch for every HSRP instance. When specifying HSRP timers, In the A CUSTOMER design
single supervisors will be used, however in the case that vPC dual supervisors are used in the future, the
recommendation is to configure and extended hold timer on all devices to support NSF during ISSU and
supervisor switchovers sub second hellos should not be used. Values of 1s hello and 3s hold timers are
ideal. Lastly IP redirects should be disabled and MD5 authentication is recommended.
Shown below is an example NX-OS configuration for HSRP on SVI interfaces:
Primary
N5K-1(config)# feature hsrp
N5K-1(config)# feature interface-vlan
N5K-1(config)# hsrp timers extended-hold 15
N5K-1(config)# interface Vlan10

N5K-1(config-if)# no ip redirects
N5K-1(config-if)# hsrp version 2
N5K-1(config-if-hsrp)# hsrp 10
N5K-1(config-if-hsrp)# authentication md5 key-string <text string>
N5K-1(config-if-hsrp)# preempt delay minimum 30
N5K-1(config-if-hsrp)# priority 110
N5K-1(config-if-hsrp)# timers 1 3
N5K-1(config-if-hsrp)# ip <hsrp address>
Secondary
N5K-2(config)# feature hsrp
N5K-2(config)# feature interface-vlan
N5K-2(config)# hsrp timers extended-hold 15
N5K-2(config)# interface Vlan10

N5K-2(config-if)# no ip redirects
N5K-2(config-if)# hsrp version 2
N5K-2(config-if-hsrp)# hsrp 10
N5K-2(config-if-hsrp)# authentication md5 key-string F1r$t_hop
N5K-2(config-if-hsrp)# preempt delay minimum 180
N5K-2(config-if-hsrp)# priority 105
N5K-2(config-if-hsrp)# timers 1 3
N5K-2(config-if-hsrp)# ip <hsrp address>
6.3.2 VLSM addressing using /31 Subnets for Point-to-Point links

31-bit prefixes as defined in RFC 3021 will provide twice as many subnets to be created out of the same
block of addresses as using /30 addressing and is now considered best practice for all point-to-point link. In
order to keep the new network design consistent with the existing network infrastructure, A CUSTOMER
have requested that /30 subnets are kept in used for point-to-point links.

6.3.3 Bidirectional Forwarding Detection (BFD)
Overview
Bidirectional Forwarding Detection (BFD) is a network protocol used to detect faults between
two forwarding engines connected by a link. It provides detection of faults for various physical media
type, encapsulations, topologies, and routing protocols.
BFD can enable sub-second failure detection between two adjacent devices without the need of
configuring aggressive CPU intensive protocol hello messages, this is because some of the BFD load can
be distributed onto the data plane with some I/O modules.
The key to providing sub-second failover using BFD is that it talks directly to other network protocols and
sends them a failure detection notice when network changes occur, this circumvents the delay usually
experienced by the time it takes for line protocol to go down, or the dead intervals to expire.
The current supported protocols and for BFD are:

BGP
EIGRP
OSPF
IS-IS
HSRP
PIM
Static Routes
Within the design, BFD should be used for all OSPF and PIM neighbours where Nexus and/or Catalyst
6500s are directly connected together. HSRP is not included as there are number of instances that could
cause the SVI to flap. In any case, the use of vPC reduces the need to enabled BFD on SVI interfaces as
HSRP is active/active.
BFD Asynchronous Mode

NX-OS version 5.0 and higher supports BFD in Asynchronous mode. This sends BFD control packets
between two adjacent devices to activate and maintain BFD neighbour sessions between them. You
configure BFD on both devices (or BFD neighbours). Once BFD has been enabled on the interfaces and on
the appropriate protocols, NX-OS creates a BFD session, negotiates BFD session parameters, and begins
to send BFD control packets to each BFD neighbour at the negotiated interval.
BFD Echo Mode

BFD echo mode sends echo packets from the forwarding engine to the remote BFD neighbour. The
neighbour then forwards the echo packet back along the same path in order to perform detection, the
BFD neighbour does not participate in the actual forwarding of the echo packets. The echo function and
the forwarding engine are responsible for the detection process.
You can configure BFD echo mode on one or both ends of a BFD monitored link. Echo mode slows down
the required minimum receive interval, based on the configured slow timer. The RequiredMinEchoRx BFD
session parameter is set to zero if echo mode is disabled. The slow timer becomes the required minimum
receive interval if echo mode is enabled. Echo mode is enabled by default and is the recommended
configuration.
Note: Echo mode requires that the interface participating in BFD has the no ip redirects command
applied.

BFD Considerations
The following consideration need to be made when using BFD:
For Layer 3 PortChannels and Layer 2 PortChannels used by SVI interfaces, LACP needs to be enabled.
Initial BFD Configuration

Before you can enable the BFD you need to disable the Address Identical IDS feature on NX-OS in the
default VDC. You can then enable the BFD and set the global intervals per VDC (which can be overridden at
the interface level). The recommended configuration values are shown below:
Default VDC
N7K(config)# no hardware ip verify address identical
Non-Default VDC
N7K(config)# feature bfd
N7K(config)# bfd interval 100 min_rx 100 multiplier 4
N7K(config)# bfd slow-timer 3000
BFD OSPF Configuration

BFD can be enabled globally or you can configure the interaction between BFD and EIGRP per-interface for
signalling. The configuration is shown below:
N7K(config)# router ospf 1

N7K(config-router)# bfd
OR
N7K(config)# interface Ethernet 1/1

N7K(config-if)# no ip redirects
N7K(config-if)# ip ospf bfd
BFD PIM Configuration

Once BFD has been enabled globally you can then configure the interaction between BFD and PIM for
signalling. The configuration is shown below:
N7K(config)# ip pim bfd

N7K(config-router)# bfd
N7K(config)# interface Ethernet 1/1

N7K(config-if)# no ip redirects
N7K(config-if)# ip pim bfd-instance
For other PIM configurations please refer to the Multicast Design section.

6.4 IP Services
6.4.1 DHCP Relay

DHCP relay enables and interface to forward broadcasted DHCP BOOTREQUEST packets off to a remote
DHCP server as a unicast packet, the DHCP would then respond back to the interface and the DHCPOFFER
would be broadcasted onto the segment for the client. In IOS this feature is referred to a helper address.
The configuration in NX-OS is the much the same with slightly different syntax.
An example is shown below:
N7K(config)# service dhcp

N7K(config)# interface Vlan100
N7K(config-if)# ip dhcp relay address 192.168.1.100
IOS(config)# service dhcp

IOS(config)# interface Vlan100
IOS(config-if)# ip helper-address address 192.168.1.100
6.5 Catalyst 4500-X Specifics
6.5.1 High Availability through SSO and NSF

The access areas will experience rapid failover performance through the implement of Stateful Switchover
(SSO) and Non Stop forwarding (NSF).
Cisco Catalyst 6500 series switches support fault resistance by allowing a redundant supervisor engine to
take over if the primary supervisor engine fails. Cisco NSF works with SSO to minimize the amount of time
a network is unavailable to its users following a switchover while continuing to forward IP packets.
The following events cause a switchover:

A hardware failure on the active supervisor engine
Clock synchronisation failure between supervisor engines
A manual switchover
SSO
SSO establishes one of the supervisor engines as active while the other supervisor engine is designated as
standby, and then SSO synchronizes information between them. A switchover from the active to the
redundant supervisor engine occurs when the active supervisor engine fails, or is removed from the
switch, or is manually shut down for maintenance. This type of switchover ensures that Layer 2 traffic is
not interrupted.
In networking devices running SSO, both supervisor engines must be running the same configuration so
that the redundant supervisor engine is always ready to assume control following a fault on the active
supervisor engine. SSO switchover also preserves FIB and adjacency entries and can forward Layer 3
traffic after a switchover. Configuration information and data structures are synchronized from the active
to the redundant supervisor engine at startup and whenever changes to the active supervisor engine
configuration occur. Following an initial synchronization between the two supervisor engines, SSO
maintains state information between them, including forwarding information.
During switchover, system control and routing protocol execution is transferred from the active
supervisor engine to the redundant supervisor engine. The switch requires between 0 and 3 seconds to
switchover from the active to the redundant supervisor engine. The configuration is shown in the
example below.
Switch(config)# power redundancy-mode redundant

NSF
Cisco NSF always runs in conjunction with SSO and provides redundancy for Layer 3 traffic. NSF works
with SSO to minimize the amount of time that a network is unavailable to its users following a switchover.
The main purpose of NSF is to continue forwarding IP packets following a supervisor engine switchover.
Cisco NSF is supported by the OSPF protocol for routing and is supported by Cisco Express Forwarding
(CEF) for forwarding. OSPF has been enhanced with NSF-capability and awareness, which means that
routers running these protocols can detect a switchover and take the necessary actions to continue
forwarding network traffic and to recover route information from the peer devices. Its also known as
graceful restart.
A networking device is NSF-aware if it is running NSF-compatible software. A device is NSF-capable if it

has been configured to support NSF; it will rebuild routing information from NSF-aware or NSF-capable
neighbours.
Each protocol depends on CEF to continue forwarding packets during switchover while the routing
protocols rebuild the Routing Information Base (RIB) tables. After the routing protocols have converged,
CEF updates the FIB table and removes stale route entries. CEF then updates the line cards with the new
FIB information.
Config Example:
router(conf)# router ospf <process>

router(config-router)# nsf
Note: NSF is not supported on IP base switching software
6.6 Miscellaneous Diagrams
6.6.1 Spanning Tree & FHRP

Below is a diagram that shows the recommended configuration for STP tools, STP port types and L2/L3
alignment with respect to vPC, HSRP, and STP:
Figure 4 - Spanning Tree & FHRP Diagram

Prepared for Client on 27 April 2015

Samplellddocumentv1 150427065245 Conversion Gate02

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Samplellddocumentv1 150427065245 Conversion Gate02

Uploaded by

Copyright:

Available Formats

CORE INFRASTRUCTURE MIGRATION

LOW LEVEL DESIGN

CLIENT LOGO HERE

RE-Solution Data Ltd

170 Greenford Road

Version Date Revision Name

0.1 14/02/2015 First Draft Sean Draper

0.2 Initial Peer Review

Name Company Contact Details Role / Responsibility

Darren Smith Resolution darren@re-solution.london Sales Account Manager

Tom Giembicki Resolution tom@re-solution.london CTO

Name Title Responsibility

Document Title Description

Name Title Signature

Copyright 2015 Re-Solution. All rights reserved. Page 2 of 32

Copyright 2015 Re-Solution. All rights reserved. Page 3 of 32

Figure 1 Overall Design Diagram ...................................................................................................................... 10

Table 1 - Device Abbreviations ............................................................................................................................. 9

Copyright 2015 Re-Solution. All rights reserved. Page 4 of 32

Copyright 2015 Re-Solution. All rights reserved. Page 5 of 32

Copyright 2015 Re-Solution. All rights reserved. Page 6 of 32

1.2 Design Scope

Design of the Enterprise Core as well as the Nexus switches

Example of best practice configuration

Enterprise 4500 and Nexus device security

Enterprise 4500 and Nexus device management

The following are considered out of scope of this document:

Fibre optic cabling design

Power provisioning for DC and communication rooms

1.3 Document Structure

Cisco Catalyst 4500 Hardware

Nexus 5000 Hardware

Network Management & Monitoring

Copyright 2015 Re-Solution. All rights reserved. Page 7 of 32

The network hardware in the new design is as follows:

2.2 Core and Server Aggregation

2.3 Server Access Layer (FlexPod)

Copyright 2015 Re-Solution. All rights reserved. Page 8 of 32

2.4 Device Abbreviations

CAL Core Aggregation Layer

Table 1 - Device Abbreviations

2.5 Example Configurations

Copyright 2015 Re-Solution. All rights reserved. Page 9 of 32

Catalyst 4500X Series VSS Catalyst 4500X Series

UCS B200 M3 UCS B200 M3

2x 10 GbE with 2x 10 GbE with

FCoE per FI SLOT

Drive Slot Cover

OK FAIL OK FAIL OK FAIL OK FAIL

UCS B200 M3 UCS B200 M3

FCoE per FI SLOT

Drive Slot Cover

OK FAIL OK FAIL OK FAIL OK FAIL

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB

600GB 600GB 600GB 600GB