BRKDCT 2610

Next Generation Data Centre Architecture
BRKDCT-2610
Reference Sessions
BRKDCT-2023 - Evolution of the Data Centre Access Architecture
BRKDCT-2011 - Design and Deployment of Data Centre Interconnects

BRKVIR-2006 - Deployment of VN-Link with the Nexus 1000v
BRKDCT-2621 - Deploying Cisco Layer 2 Multipathing Technologies
BRKDCT-1044 - FCoE for the IP Engineer
BRKDCT-2610
2011 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Agenda
Data Centre Facilities and Network Infrastructure
Challenges and Trends
Next Generation Data Centre Technologies

Virtual Port Channels (vPC)
Fabricpath
Data Centre Interconnect
Access Layer
BRKDCT-2610
Cisco Public
Data Centre Facilities and Network

Infrastructure Challenges and Trends
BRKDCT-2610
Cisco Public
DC Environment Trends
1.
2.
Physical Infrastructure
What are the implications
1.
Power & Space
2.
Cooling and Airflow
Brownfield DCs are aging fast and are hard to retrofit!
3.
Cabling
Greendfield DCs are carefully planned, 18-24 months ahead
4.
Racks and Cabinets
Infrastructure choices affect the network architecture
Network Infrastructure
What is happening the next 24 months?
1.
Access
Migration from GE to 10GE attached servers
2.
Aggregation
Adoption of 40GE technologies: switch interconnect and servers
3.
Core
Increase Adoption of Virtualised Technology
4.
Services
Start of migration to non-STP environments: IS-IS and ECMP L2/L3
5.
Unified Environments
BRKDCT-2610
Cisco Public
Data Centre Evolution Path

Consolidation
Virtualisation
Automation
Utility
Cloud
Increase in 10 Gigabit Ethernet port density

Tighter integration between servers and the
network
Network/Server demarcation moving
inside of the server
BRKDCT-2610
Cisco Public
The Evolving Data Centre Architecture

Evolution of the Hierarchical Design Access Layer
The Data Centre Architecture has
been based on the hierarchical
switching design
Aggregation block contains the
access and aggregation layers
Dedicated service switches

provide application load
balancing, firewall, etc.
Core
Layer 3
Layer 2
Aggregation
Servers connected to 1G ports at

the access layer (both ToR and
EoR)
Services
Architecture is based on optimised

design for control plane stability and
scalability
BRKDCT-2610
Access
Cisco Public
L2 Access
Plug-and-play provisioning
Practically plug-n-play No user configuration is required to build
forwarding database
It makes it simple to support teaming or L2 multicast for clusters
Easy to segment traffic with VLANs
MAC Table
MAC Table
Layer 2
Domain
MAC Table
MAC Table
MAC Table
MAC Table
A
BRKDCT-2610
Cisco Public
Current STP Deployments

Current
STP blocks redundant uplinks
VLAN based load balancing
Loop Resolution relies on STP
BRKDCT-2610
Cisco Public
Primary
Root
Secondary
Root
Drawbacks of L2 Access
VLAN sprawl
MAC address consumption
BPDU generation is CPU intensive with increasing number of VLANs
VLAN sprawl causes flooding and broadcasts to propagate even
where they are not needed
Half of the links in the

topology are blocking
Misconfigurations can cause
Layer 2 loops which may
make switches
unmanageable
BRKDCT-2610
MAC Table
MAC Table
Layer 2
Domain
MAC Table
MAC Table
MAC Table
MAC Table
Cisco Public
10
CONS
PROS
L3 Access
Routed Access Topologies alleviate the
consumption of L2 tables
No Spanning Tree Recalcuations
L3
All links active and forwarding to

Distribution/Agg
Smaller subnets to manage and
more L3 configuration points
Difficult migration to Unified wire
topologies
IP Attached
Storage
Servers, FCoE
attached Storage
Servers
Limited VM Mobility
BRKDCT-2610
Cisco Public
11
Evolving Data Centre Architecture

Challenges for the Classical Design
Data Centre Row 1
Hypervisor based server virtualisation and
the associated capabilities (vMotion, )
are changing multiple aspects of the Data
Centre design
How large do we need to scale Layer 2?
Where does the storage fabric exist (NAS,
SAN, )
Data Centre Row 2
How much capacity does a server need

Where is the policy boundary (security,
QoS, WAN acceleration, )?
Where and how do you connect the

servers?
BRKDCT-2610
Cisco Public
12
Current Challenges in the Virtualised Data Centre
Provisioning of network services for VMs (Port profiles, etc.)
Coordination of VM migration
Lack of visibility of VM to VM traffic
Deployment of advanced functionality down to the VMs (ACLs, security, etc.)
Scaling management applications to match growth in deployed switches
Lack of common management tools
Difficulty in segregating server and network management functions
BRKDCT-2610
Cisco Public
13
Next Generation Data Centre Technologies
BRKDCT-2610
Cisco Public
14

Evolution of the Hierarchical Design Access Layer
The Data Centre Architecture has
been based on the hierarchical
switching design
Aggregation block contains the
access and aggregation layers
Dedicated service switches

provide application load
balancing, firewall, etc.
Core
Layer 3
Layer 2
Aggregation
Servers connected to 1G ports at

the access layer (both ToR and
EoR)
Services
Architecture is based on optimised

design for control plane stability and
scalability
BRKDCT-2610
Access
Cisco Public
15
Virtual Port-Channels (vPC)
BRKDCT-2610
Cisco Public
16
Virtual Port Channel - vPC

vPC is a Port-channeling concept extending link
aggregation to two separate physical switches
Allows the creation of resilient L2 topologies
based on Link Aggregation.
Eliminates the need for STP in the accessdistribution
Physical Topology
Logical Topology
Virtual Port Channel
Provides increased bandwidth
L2
All links are actively forwarding
Si
Si
vPC maintains independent control planes
vPC switches are joined together to form a

domain
vPC domain
BRKDCT-2610
Cisco Public
vPC
Non-vPC
Increased BW with vPC
17
Virtual Port Channel vPC

vPC allows a single device to use a
port channel across two neighbour
switches (vPC peers)
vPC Peers
Eliminate STP blocked ports

Portchannel
vPC Peers
Layer 2 port channel only

Provide fast convergence upon
link/device failure
BRKDCT-2610
Port
channel
Cisco Public
18
vPC and Spanning-Tree

STP for vPCs is controlled by
the vPC operationally primary switch and
only such device sends out BPDUs on STP
designated ports.
This happens irrespectively of where the
designated STP Root is located
Primary
vPC (root)
Secondary
vPC
BPDUs
The vPC operationally secondary device

proxies STP BPDU messages from access
switches toward the primary vPC
BRKDCT-2610
Cisco Public
19
vPC Peer Switch

vPC Primary
vPC Peer-link
S1
Root
Root
Peer-switch
S5
S2
Root
Peer-switch
vPC2
S3
S4
Physical representation
vPC Secondary
S2
BPDU
vPC1
vPC Primary
Root
S1
vPC Secondary
S6
S5
S3
S4
Logical representation
S6
The two vPC peers send the same information:

they look like a single root bridge
vPC Peer-link is a regular STP link; it is always in FWD state for all
vPC VLAN
BRKDCT-2610
Cisco Public
20
Virtual Port Channel - vPC

vPC Control Plane - FHRP
HW Programmed to forward frames

sent to the FHRP MAC address on
BOTH Switches
vPC maintains dual active control planes and

STP still runs on both switches
HSRP Active
HSRP Standby
HSRP active process communicates the

active MAC to its neighbour
Only the HSRP active process responds to
ARP requests
HSRP active MAC is populated into the L3
hardware forwarding tables, creating a local
forwarding capability on the HSRP standby
device
Consistent behaviour for HSRP, VRRP and
GLBP
No need to configure aggressive FHRP hello
timers as both switches are active
BRKDCT-2610
Cisco Public
21
Layer 3 and vPC Designs

Layer 3 and vPC Design
Use L3 links to hook up routers and peer with a vPC domain
Dont use L2 port channel to attach routers to a vPC domain unless you statically
route to HSRP address
If both, routed and bridged traffic is required, use individual L3 links for routed traffic
and L2 port-channel for bridged traffic
Switch
Switch
Po2
Po2
P
7k1
L3
ECMP
7k2
Po1
P
BRKDCT-2610
Router
Routing Protocol Peer

Dynamic Peering
Relationship
Cisco Public
Router
22
ASA with LACP Support

ASA
ASA
vPC80
vPC40
standby
active
vPC10
vPC11
NX5K
ASA keep-alive and FO link
BRKDCT-2610
Cisco Public
23
FabricPath
BRKDCT-2610
Cisco Public
24
L2 Requires a Tree
Branches of trees never
interconnect (no loop)
5 Logical Links
11 Physical Links
S2
S1
S3
Spanning Tree Protocol (STP) typically used to build this tree

Tree topology implies:
Wasted bandwidth increased oversubscription
Sub-optimal paths
Conservative convergence (timer-based)
BRKDCT-2610
Cisco Public
25
Existing L2 Technology Is Not Perfect

Even in a vPC topology, the design is less flexible than a routed
topology, because its not possible to distribute traffic among more
than 2 aggregation devices
Loops are still possible; this is not a problem in routed networks.
Layer 2 tables are not used efficiently, flooding causes l2 tables to be

populated with unnecessary MAC addresses
Is it possible to bring the advances of L3 into the world of L2?
BRKDCT-2610
Cisco Public
26
Cisco FabricPath
Scaling and Simplifying Layer 2 Ethernet Networks
Up to 16 Agg
switches
160+ Tbps
switching capacity
Up to 32
access switches
Cisco FabricPath Network
-All Links Active
Traditional Spanning Tree Based Network

-Blocked Links
Eliminate Spanning tree limitations
Multi-pathing across all links, high cross-sectional bandwidth
High resiliency, faster network re-convergence
Any VLAN, any where in the fabric eliminate VLAN Scoping
BRKDCT-2610
Cisco Public
27
The Layer 2 Evolution

Spanning-Tree
Active Paths
POD
Bandwidth
vPC
FabricPath
Single
Dual
16 Way
Up to 10 Tbps
Up to 20 Tbps
Up to 160 Tbps
Layer 2 Scalability
Infrastructure Virtualisation and Capacity
BRKDCT-2610
Cisco Public
28
Cisco FabricPath Overview

Cisco FabricPath
Data Plane Innovation
Control Plane Innovation
No MAC learning via flooding
Plug-n-Play Layer 2 IS-IS
Routing, not bridging
Support unicast and multicast
Built-in loop-mitigation
Fast, efficient, and scalable
BRKDCT-2610
Time-to-Live (TTL)
Equal Cost Multipathing (ECMP)
RPF Check
VLAN and Multicast Pruning
Cisco NX-OS
Cisco Nexus Platform
Cisco Public
29
FabricPath Feature Set
16-Way Equal Cost Multipathing (ECMP) at Layer 2

FabricPath Header
Hierarchical addressing with built in loop mitigation (RPF,TTL)
Conversational MAC Learning
Efficient use of hardware resource by learning only MACs for interested
hosts
Up to
16 Switches
Interoperability with existing classic

Ethernet networks
VPC + allows VPC into a L2 Fabric
STP Boundary Termination

FabricPath
Multi-Topology providing traffic

engineering capabilities
BRKDCT-2610
Access Switches
Cisco Public
30
Data Plane Operation

Encapsulation to creates hierarchical address scheme
FabricPath header is imposed by ingress switch
Ingress and egress switch addresses are used to make Routing decision
No MAC learning required inside the L2 Fabric
S42
S11
C
A
FabricPath
Header
FabricPath Routing
S11 S42
DATA
S11
STP
FabricPath
Domain
Ingress Switch
S42
Egress Switch
AC
C
A
DATA
BRKDCT-2610
STP Domain 1
A
C
Cisco Public
STP Domain 2
AC
L2 Bridging
31
Control Plane Operation

Plug-N-Play L2 IS-IS - used to manage forwarding topology
Assigned switch addresses to all FabricPath enabled switches automatically
(no user configuration required)
Compute shortest, pair-wise paths
Support equal-cost paths between any FabricPath switch pairs
S1
S2
S3
S4
FabricPath Routing Table

Switch
IF
S1
L1
S2
L2
S3
L3
S4
L4
S12
L1, L2, L3, L4
S42
L1, L2, L3, L4
BRKDCT-2610
L1
L2
S11
L3
L4
S12
Cisco Public
L2 Fabric
S42
32
Unicast with FabricPath

Forwarding decision based on FabricPath Routing Table
Support more than 2 active paths (up to 16) across the Fabric
Increase bi-sectional bandwidth beyond port-channel
High availability with N+1 path redundancy
S1
Switch
IF
S42
L1, L2, L3, L4

MAC
IF
1/1
S42
L1
S2
L2
S4
L3
L4
S11
S3
S12
L2 Fabric
S42
1/1
BRKDCT-2610
Cisco Public
33
Multicast with FabricPath

Forwarding through distinct Trees
Several Trees are rooted in key location inside the fabric
All Switches in L2 Fabric share the same view for each Tree
Multicast traffic load-balanced across these Trees
Root for
Tree #1
Root for
Tree #2
Ingress switch for

FabricPath decides which
tree to be used and add
tree number in the header
L2 Fabric
BRKDCT-2610
Cisco Public
34
Loop Mitigation with FabricPath

Minimise impact of transient loop with TTL and RPF Check
Root
STP Domain
S1
Root
S2
TTL=2
TTL=1
L2 Fabric
S10
TTL=3
TTL=0
Block redundant paths to ensure

loop-free topology
Frames loop indefinitely if STP
failed
Could results in complete network
melt-down as the result of flooding
BRKDCT-2610
TTL is part of FabricPath header

Decrement by 1 at each hop
Frames are discarded when TTL=0
RPF check for multicast based on
tree info
Cisco Public
35
VL30
VL10
VLAN Pruning in L2 Fabric

Switches indicate locally
interested VLANs to the rest of
the L2 Fabric
Shared
Broadcast Tree
VLAN 10
L2 Fabric
BRKDCT-2610
Broadcast traffic for any VLAN only

sent to switches that have
requested for it
VL20
VL10
VL20
VL30
L2 Fabric
VLAN 20
L2 Fabric
VLAN 30
L2 Fabric
Cisco Public
36
vPC+ Enhancement for FabricPath

MAC Table
MAC Table
A S4
A ???
S3
S3
L2 Fabric
L2 Fabric
S3 S1 B
S3 S2 B
Payload
vPC
S1
S3 S4 B
Payload
vPC+
S2
Payload
S3 S4 B
S1
MAC Table
B S3
B S3
Payload
S2
MAC Table
Payload
S4
A
B
For Switches at L2 Fabric Edge
Payload
A
vPC is still required to provide

active/active L2 paths for dual-homed CE
devices or clouds
Each vPC domain is represented by an

unique Virtual Switch to the rest of L2
Fabric
However, MAC Table only allows 1-to-1

mapping between MAC and Switch ID
Switch ID for such Virtual Switch is then

used as Source in FabricPath encapsulation
BRKDCT-2610
Cisco Public
37
Migration from vPC to vPC+

1. Peer-link & all vPCs must be on F1 ports
2. Add fabricpath virtual switch ID under the VPC domain config on each switch
(this is disruptive, all VPCs will flap).

3. Configure the VPC+ peer-link as "switchport mode fabricpath".
The vPC+ PL will not learn/synchronise anymore MAC@ across the PL.
4. Previous configuration for vPC (vPC member ports) remain the same.
5. Previous configuration for FHRP remain the same
6. Change VLAN from CE mode to FP mode (maybe this would be the first step
of migration)
BRKDCT-2610
Cisco Public
38
Connect L3 or Services to L2 Fabric

Layer 3 Network
L2
L3
FHRP
L2 Fabric
Multi-pathing
L3
FHRP
Active
FabricPath enables multipathing for

bridged traffic
However, FHRP allows only 1 active
gateway for each host, therefore prevent
traffic that needs to be routed to take
advantage
of multi-pathing
BRKDCT-2610
L2
FHRP
Active
FHRP
L2 Fabric
Multi-pathing
Layer 3 Network
Provide active/active data-plane for

FabricPath with no change to existing
FHRP
Allow multi-pathing even for routed traffic

Same feature can be leveraged by
service nodes as well
Cisco Public
39
STP Boundary Termination

FabricPath
(L2 IS-IS)
L2 Fabric
Classical
Ethernet
(STP)
STP
Domain
2
STP
Domain 1
FabricPath Port
CE Port
L2MP Core is presented as a single bridge to all connected CE devices

STP BPDUs are processed and terminated by CE Ports
CE devices not interconnected will form separate STP domains
Loops outside L2 Fabric will be blocked within each STP domain
L2 Fabric should be the root for all connected STP domain. CE ports will be put into
blocking state when superior BPDU is received
BRKDCT-2610
Cisco Public
40
FabricPath Configuration
No L2 IS-IS configuration required
New feature-set keyword introduced to allow multiple conditional services
required by FabricPath to be enabled in one shot
Simplified operational model only 3 CLIs to get FabricPath up and running

N7K(config)# feature-set fabricpath
N7K(config)# vlan 10-19
N7K(config-vlan)# mode fabricpath
N7K(config)# interface e1/1
N7K(config-if)# switchport mode
fabricpath
L2 Fabric
BRKDCT-2610
Cisco Public
FabricPath Port
41
CE Port

MAC learning method designed to conserve MAC table entries on FabricPath
edge switches
FabricPath core switches do not learn MACs at all
Each forwarding engine distinguishes between two types of MAC entry:

Local MAC MAC of host directly connected to forwarding engine
Remote MAC MAC of host connected to another forwarding engine or switch
Forwarding engine learns remote MAC only if bidirectional conversation

occurring between local and remote MAC
MAC learning not triggered by flood frames
Conversational learning enabled in all FabricPath VLANs

BRKDCT-2610
Cisco Public
42

FabricPath
MAC Table on S300
FabricPath
MAC Table on S100
MAC
MAC
IF/SID
S200 (remote)
e7/10 (local)
S300
MAC C
S100
IF/SID
e1/1 (local)
S200 (remote)
FabricPath
MAC Table on S200
FabricPath Core
S200
MAC A
MAC
IF/SID
S100 (remote)
e12/1(local)
S300 (remote)
MAC B
BRKDCT-2610
Cisco Public
43
Transparent Interconnection of Lots of

Links (TRILL) and Fabric Path
Fabricpath
TRILL
Yes
Yes
Inter-switch links
Point-to-point only
Point-to-point OR shared
Emulated switch
Yes
No
FHRP active/active
(AnyCast FHRP in the future)
Yes
No
Multiple topologies
Yes
No
Conversational learning
Yes
No
Frame routing
(ECMP, TTL, RPFC etc)
BRKDCT-2610
Cisco Public
44
FabricPath Summary
FabricPath is simple, keeps the attractive aspects of Layer 2
Transparent to L3 protocols
No addressing, simple configuration and deployment
FabricPath is scalable
Can extend a bridged domain without extending the risks generally associated to
Layer 2 (frame routing, TTL, RPFC)
FabricPath is efficient
High bi-sectional bandwidth (ECMP)
Optimal path between any two nodes
BRKDCT-2610
Cisco Public
45
Data Centre Interconnect (DCI)
BRKDCT-2610
Cisco Public
46
Network port
Edge or portfast port type
Normal port type
BPDUguard
BPDUfilter
Rootguard
E
-
Multi-layer vPC for Agg and DCI
B
F
R
DC 1
AGGR
N
-
- R
R
N
vPC domain 10
vPC domain 20
Key Recommendations
E
B
vPC Domain id for facing vPC layers should be different

No Bridge Assurance on interconnecting vPCs
BPDU Filter on the edge devices to avoid BPDU propagation
No L3 peering between DCs (i.e. L3 over vPC)
Server Cluster
E
B
ACCESS
ACCESS
BRKDCT-2610
AGGR
DC 2
vPC domain 21
Long Distance
CORE
CORE
vPC domain 11
Server Cluster
Cisco Public
47

Encrypted Interconnect
DC-1
Nexus 7010
DC-2
vPC
Nexus 7010
vPC
CTS Manual Mode

(802.1AE 10GE line-rate
encryption)
No ACS is required
Nexus 7010
BRKDCT-2610
Nexus 7010
Cisco Public
48
Overlay Transport Virtualisation (OTV)

Ethernet LAN Extension over
any Network
Ethernet in IP MAC routing

Multi-Data Centre scalability
Simplified Configuration &
Operation
Seamless overlay - No network
re-design
High Resiliency
Failure domain isolation
Seamless Multi-homing
Maximises available
bandwidth
Automated multi-pathing
Optimal multicast replication
Single touch site configuration

BRKDCT-2610
Cisco Public
49
OTV Interface Types

Edge Device
Internal Interfaces
OTV
External Interface
Overlay
Interface
Overlay Interface
L2 L3
Internal
Interfaces
BRKDCT-2610
Cisco Public
Join
Interface
Core
50
OTV Topology Discussion
BRKDCT-2610
Cisco Public
51
Egress Routing Localisation

FHRP Filtering Solution
Filter FHRP with combination of VACL and MAC route filter
Result: Still have one HSRP group with one VIP, but now have active
router at each site for optimal first-hop routing
HSRP Hellos
HSRP Hellos
HSRP Filtering
HSRP
Active
HSRP
Active
HSRP
Standby
HSRP
Standby
ARP
reply
ARP for
HSRP VIP
VLAN
20
BRKDCT-2610
Cisco Public
VLAN
10
52
Routing Based Ingress Optimisation

VM IP Address
10.10.10.1
LISP
1
IP_DA = 10.10.10.1
IP_DA = 10.10.10.1
ETR
IP_DA = 10.10.10.1
Ingress Tunnel
Router (ITR)
6
Encap
IP_DA = B
Prefix
(EID)
ISP A
Data Centre A
3
Decap
10.10.10.1
A,
B
Moved
10.10.10.2
A, B
10.10.10.5
C, D
10.10.10.6
C, D
Agg
IP_DA = C
IP_DA = 10.10.10.1
Route Locator
(RLOC)
ISP B
Data Centre B
to C, D
ETR
DecapIP_DA = 10.10.10.1
Agg
LAN Extension
Access
Access
4
VM= 10.10.10.1
BRKDCT-2610
Default GW = 10.10.10.100
Cisco Public
VM= 10.10.10.1
Default GW = 10.10.10.100
53
Access Layer
BRKDCT-2610
Cisco Public
54
What Is FEX
FEX is an extension of the
switch that it connects to.
Nexus 5000 and Nexus 7000

can be extended with a Nexus
2000
FEX can be connected with
1/3/5/7/10m CX1, SR, LR, FET
FEX inherits the features of the
device it is connected to
BRKDCT-2610
Cisco Public
55
Nexus 2000 Designs

Nexus 5000 Topologies (Nexus 2248TP & 2232PP)
Straight Through
vPC Supported
with up to 2 x 8
links
Dual Homed
FCoE Adapters
supported on 10G
N2K interfaces
Redundancy model Dual Switch with redundant

fabric
Provides isolation for Storage topologies (SAN A
and B)
Port Channel and Pinning supported for Fabric
Link
BRKDCT-2610
Local
Etherchannel
with up to 8
links
Redundancy model Single switch with dual

supervisor for fabric, data control & management
planes
No SAN A and B isolation (VSAN isolation
sufficient in the future?)
Cisco Public
56
Nexus 2000 Design

Nexus 7000 Topologies (Nexus 2248TP & 2232PP)
Nexus 2248TP & 2232PP
NIC Teaming:
TLB/ALB
Local
Etherchannel
with up to 8
links
Fabric links supported on N7K-M132XP-12

& N7K-M132XP-12L
BRKDCT-2610
Local port channel support on 2248 & 2232

(Future release)
No support for DCB and FCoE (parent
switch fabric ports not DCB capable yet)
Cisco Public
57
Nexus 2000 Design

Topologies Next Steps
Nexus 5000
Future
Nexus 7000 vPC
MCEC
Etherchannel
with up to 16
links
MCEC
Etherchannel
with up to 16
links
Redundancy model Dual Switch (each

switch supports redundant supervisors)
Future release
BRKDCT-2610
Redundancy model Single switch with

dual supervisor, fabric, line card, data
control & management planes
Cisco Public
58
Current Data Centre Architecture

Where Is the Edge?
The Data Centre Edge has historically
been well defined from a technical
and operational perspective
There have always been exceptions
to this rule but they were usually
special cases and often involved
dedicated access layer designs
Eth
2/12
FC
3/11
Edge of the Network and Fabric

NIC
HBA
PCI-E Bus
The location of the edge is moving

Hypervisor Virtual Switches
Operating
System and
Device Drivers
SR-IOV
FCoE
BRKDCT-2610
Cisco Public
59

Technology Disruptor - Virtualisation
Virtualised
Traditional
App
App
App
OS
OS
OS
App
App
App
OS
OS
OS
1 Application
Transition
App
App
App
OS
OS
OS
App
App
App
OS
OS
OS
App
App
App
OS
OS
OS
Many Apps,
or VMs
...1 Server,
or Host
...1 Server
20,000,000
17,500,000
Tipping Point
15,000,000
12,500,000
10,000,000
7,500,000
5,000,000
2,500,000
0
2005
BRKDCT-2610
2006
2007
2008
2009
2010
Virtualized
Non-Virtualized
Cisco Public
2011
2012
2013
2014
Source: IDC, Nov 2010
60
Current Data Centre Architecture

Hypervisor vSwitchWhere Is the Edge?
Hypervisor based compute virtualisation
moves the edge of
the Fabric
Eth
2/12
PCI-E bus and storage and network

connectivity resources are virtualised
FC
3/11
HBA
pNIC
vSwitch
VMFS (VMWare)
PCI-E Bus
NPV (provides FC SAN virtualisation)

VETH
With a shift in the edge of the fabric

comes a change in the operational
practices and fabric design requirements
BRKDCT-2610
Cisco Public
VMFS
SCSI
Edge of the
Fabric
VNIC
Hypervisor provides
virtualisation of PCI-E
resources
61
Unified Fabric
IEEE DCB
Developed by IEEE 802.1 Data Centre Bridging Task Group (DCB)
Standard / Feature
Status of the Standard
IEEE 802.1Qbb
Priority-based Flow Control (PFC)
Done! And we are compliant!
IEEE 802.3bd
Frame Format for PFC
Done! And we are compliant!
IEEE 802.1Qaz
Enhanced Transmission Selection (ETS) and
Data Centre Bridging eXchange (DCBX)
Just completed WG; mid-March 2011
IEEE 802.1Qau Congestion Notification
Done!
IEEE 802.1Qbh Port Extender
In first working group ballot (which is the next phase after

successful task group ballot as indicated in the
slide). Expect to complete in 6-12 months.
CEE (Converged Enhanced Ethernet) is an informal group of

companies that submitted initial inputs to the DCB WGs.
BRKDCT-2610
Cisco Public
62
Priority Flow Control

Fibre Channel over Ethernet Flow Control
Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1p
When link is congested, CoS assigned to FCoE will be PAUSEd so traffic will not be
dropped
Other traffic assigned to other CoS will continue to transmit and rely on upper layer
protocols for retransmission
Transmit Queues
Fibre Channel
Ethernet Link
One
One
Two
Three
BRKDCT-2610
R_RDY
Packet
B2B Credits
Receive Buffers
Two
PAUSE
STOP
Three
Four
Four
Five
Five
Six
Six
Seven
Seven
Eight
Eight
Cisco Public
Eight
Virtual
Lanes
63
Enhanced Transmission Standard

Bandwidth Management -- IEEE 802.1Qaz
Required when consolidating I/O Its a QoS problem
Prevents a single traffic class of hogging all the bandwidth and starving other
classes
When a given load doesnt fully utilise its allocated bandwidth, it is available to
other classes
Helps accommodate for classes of a burtsy nature
Offered Traffic
3G/s
BRKDCT-2610
3G/s
10 GE Link Realised Traffic Utilisation
2G/s
3G/s
3G/s
3G/s
3G/s
4G/s
6G/s
t1
t2
t3
Cisco Public
3G/s
HPC Traffic
3G/s
2G/s
3G/s
Storage Traffic
3G/s
3G/s
3G/s
LAN Traffic
4G/s
5G/s
t1
t2
t3
64
Data Centre Bridging eXchange

Control Protocol the handshake
Negotiates Ethernet capabilitys : PFC, ETS, CoS values between

DCB capable peer devices
Simplifies Management : allows for configuration and distribution
of parameters from one node to another
Responsible for Logical Link Up/Down signalling of Ethernet and

Fibre Channel
DCBX negotiation failures result in:
per-priority-pause not enabled on CoS values

vfc not coming up when DCBX is being used in FCoE
environment
BRKDCT-2610
Cisco Public
65
Fibre Channel over Ethernet

What enables it?
10Gbps Ethernet
Lossless Ethernet
Matches the lossless behaviour guaranteed in FC by B2B credits
Ethernet jumbo frames

Max FC frame payload = 2112 bytes
FCS
EOF
FC Payload
CRC
FC
Header
FCoE
Header
Ethernet
Header
Normal ethernet frame, ethertype = FCoE

Same as a physical FC frame
Control information: version, ordered sets (SOF, EOF)

BRKDCT-2610
Cisco Public
66
FCoE Building Blocks
The New BuzzwordUnified

Unified I/O using Ethernet as the transport medium in all network
environments -- no long needing separate cabling options for LAN
and SAN networks
Shared Wire a single DCB Ethernet link actively carrying both
LAN and Storage (FC/FCoE/NAS/iSCSI) traffic simultaneously
Dedicate Wire -- a single DCB Ethernet link capable of carrying

all traffic types but actively dedicated to a single traffic type for
traffic engineering purposes
Unified Fabric An Ethernet Network made up of Unified Wires
everywhere: all protocols network and storage transverse all links
simultaneously
BRKDCT-2610
Cisco Public
67
Fibre Channel over Ethernet Port Types

Fibre Channel over Ethernet Switch
FCF VE_Port
Switch
VE_Port
VF_Port
FCoE_
VNP_Port NPV
Switch
VF_Port
VN_Port End
Node
VF_Port
VN_Port End
Node
FCoE Switch : FCF

BRKDCT-2610
Cisco Public
68
Unified Fabric Design

Unified Edge
The first phase of the Unified Fabric evolution
design focused on the fabric edge
FC
LAN Fabric
Unified the LAN Access and the SAN Edge by

using FCoE
Consolidated Adapters, Cabling and Switching at
the first hop in the fabrics
Fabric A
FCoE
The Unified Edge supports multiple LAN and

SAN topology options
Fabric B
FC
Virtualized Data Centre LAN designs

Fibre Channel edge with direct attached
initiators and targets
Nexus 5000
FCF Switch Mode
Nexus 5000
FCF NPV Mode
Fibre Channel edge-core and edge-coreedge designs

Fibre Channel NPV edge designs
The Unified Edge

BRKDCT-2610
Cisco Public
69
Unified Fabric Design
Nexus Edge participates in

both distinct FC and IP Core
topologies
Unified Edge
Converged Network Adapter (CNA) presents
two PCI address to the Operating System (OS)
OS loads two unique sets of drivers and

manages two unique application topologies
Nexus 5000
FCF-B
Unified Wire
shared by both
FC and IP
topologies
Link
ETH
FC
FC Driver
bound to FC
HBA PCI
address
NIC Teaming provides failover within the

same fabric (VLAN)
Nexus Unified
Edge supports
both FC and IP
topologies
10GbE
SAN Multi-Pathing provides failover

between two fabrics (SAN A and SAN
B)
10GbE
Server participates in both topologies since it

has two stacks and thus two views of the same
unified wire
Nexus 5000
FCF-A
Ethernet Driver
bound to
Ethernet NIC PCI
address
PCIe
Fibre Channel
Drivers
Ethernet
Drivers
Operating System
BRKDCT-2610
Cisco Public
70
Unified Fabric with FCoE

FCoE Design
SAN A
SAN B
A VLAN is dedicated for every VSAN in the fabric

The VLAN is signaled to the hosts over FIP
The FCoE controller in the host tags all
subsequent FIP login and FCoE frames with the
signaled FCoE VLAN
VSAN 2
This doesnotrequire trunking to be enabled at

the host driver as tagging is performed by the
CNA
VLAN 10,20
All ports in the FCoE network have to be enabled

for trunking to be able to carry VLAN tagged
frames
STP Edge Trunk

VLAN 10,30
Isolated Edge switches for SAN A and B and

separate LAN switches for NIC 1 and NIC 2
(standard NIC teaming)
BRKDCT-2610
FCF
FCF
! VLAN 20 is dedicated for VSAN 2 FCoE traffic

(config)# vlan 20
(config-vlan)# fcoevsan2
Cisco Public
71
Unified Fabric with FCoE

FCoE Design
SAN A
MCEC results in diverging LAN and SAN high

availability topologies
FC maintains separate SAN A and SAN
B topologies
vPC Peer Link

VLAN 10 ONLY
HERE!
LAN utilises a single logical topology

In vPC enabled topologies in order to ensure
correct forwarding behaviour for SAN traffic
specific design and forwarding rules must be
followed
N5K1
N5K2
vPC Peers
VLAN 10,20
While the port-channel is the same on N5K-1

and N5K-2, the FCoE VLANs are different
STP Edge Trunk

MCEC for IP Only VLAN 10
vPC configuration works with Gen-2 FIP

enabled CNAs ONLY
VLAN 10,30
FCoE VLANs are not carried on the vPC

peer-link
FCoE and FIP ethertypes are not forwarded

over the vPC peer
link
BRKDCT-2610
SAN B
Direct Attach vPC Topology

Cisco Public
72
Virtual Expansion Ports (VE_Ports)

FC
FCoE
Creates a standards based FCoE ISL

No further standards or protocols necessary
for implementing multihop FCoE
E
VE
VE
Scalable Solution
Supports up to 7 hops same as FC
10,000 logins per fabric same as FC
E_Ports
with FC
FCoE
BRKDCT-2610
VE_Ports
with FCoE
Cisco Confidential
Cisco Public
73
FCoE Multi-Tier Fabric Design

Extending FCoE past the Unified Edge
LAN Fabric
Extending FCoE Fibre Channel fabrics beyond

direct attach initiators can be achieved in two
basic ways
Fabric A
Fabric B
FCF
Extend the Unified Edge
VE
Using FCoE
for ISL
between FC
VE
Switches
Add DCB enabled Ethernet switches

between the VN and VF ports (stretch the
link between the VN_Port and the VF_Port)
Extend Unified Fabric capabilities into the SAN
Core
FCF
Switch Mode
VF
Leverage FCoE wires between Fibre

Channel switches (VE_Ports)
What design considerations do we have when
extending FCoE beyond the edge?
VN
High Availability
Oversubscription for SAN and LAN

Ethernet layer 2 and STP design
BRKDCT-2610
Cisco Public
Extending
FCoE into a
multi-hop
Ethernet
Access
Fabric
DCB + FIP
Snooping
Bridge
74
Fibre Channel Aware Device

FCoE NPV
What does an FCoE-NPV device do?
FCoE NPV bridge" improves over a "FIP
snooping bridge" by intelligently proxying FIP
functions between a CNA and an FCF
Active Fibre Channel forwarding and security
element
FCF
FCoE-NPV load balance logins from the CNAs

evenly across the available FCF uplink ports
FCoE NPV will take VSAN into account when
mapping or pinning logins from a CNA to an
FCF uplink
Emulates existing Fibre Channel Topology (same
mgmt, security, HA)
Avoids Flooded Discovery and Configuration (FIP)
BRKDCT-2610
Cisco Public
Fibre Channel Configuration

and Control Applied at the
Edge Port
Proxy FCoE VLAN
Discovery
VF
VNP
FCoE
NPV
Proxy FCoE FCF

Discovery
75
FCoE Multi-Tier
Larger Fabric Multi-Hop Topologies
N7K or MDS FCoE
enabled Fabric
Switches
Multi-hop edge/core/edge topology

Core SAN switches supporting FCoE
N7K with DCB/FCoE line cards
MDS with FCoE line cards (Sup2A)
VE
VF
VE
Edge FC switches supporting either

N5K - E-NPV with FCoE uplinks to
the FCoE enabled core (VNP to
VF)
VE
VNP
Edge FCF
Switch
Mode
VE
N5K or N7K - FC Switch with

FCoE ISL uplinks (VE to VE)
Scaling of the fabric (FLOGI, ) will

most likely drive the selection of which
mode to deploy
BRKDCT-2610
Servers, FCoE
attached Storage
Cisco Public
Servers
FC Attached
Storage
Edge Switch
in E-NPV
Mode
76
Cisco Nexus 1000V Components

Virtual Ethernet Module(VEM)
Virtual Supervisor Module(VSM)
BRKDCT-2610
CLI interface into the Nexus 1000V
Replaces Vmwares virtual switch
Leverages NX-OS
Controls multiple VEMs as a single

network device
Enables advanced switching capability

on the hypervisor
Provides each VM with dedicated
switch ports
Cisco Public
vCenter Server
77
Port Profile: Network Admin View

n1000v# show port-profile name WebProfile
port-profile WebServers-PP
description:
status: enabled
capability uplink: no
system vlans:
port-group: WebServers
config attributes:
switchport mode access
switchport access vlan 110
no shutdown
evaluated config attributes:
switchport mode access
switchport access vlan 110
no shutdown
assigned interfaces:
Veth10
BRKDCT-2610
Cisco Public
Support Commands Include:

Port management
VLAN
PVLAN
Port-channel
ACL
Netflow
Port Security
QoS
78
Port Profile: Server Admin View
BRKDCT-2610
Cisco Public
79
Connectivity Best Practices

VM
VM
VM
VM
If the upstream switch can be clustered

(VPC, VBS Stack, VSS) use LACP
vSphere
VM
VM
VM
VM
vSphere
If the upstream switch can NOT be

clustered use MAC-PINNING
BRKDCT-2610
Cisco Public
80
What is vPath ?
vPath
Nexus 1000V- VEM

vPath is intelligence build into Virtual Ethernet Module (VEM) of
N1KV (1.4 and above)
vPath has two main functions:
a. Intelligent Traffic Steering to VSG
b. Offload the processing from VSG to VEM

vPath is Multitenant Aware
Leveraging vPath enhances the service performance by moving

the processing to Hypervisor
BRKDCT-2610
Cisco Public
81
Virtual Security Gateway

Intelligent Traffic Steering with vPath
VM
VM
VM
VM
VM
VM
VM
VM
VM
VNMC
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Nexus 1000V
vPath
Distributed Virtual Switch
Decision
Caching
1
BRKDCT-2610
Initial Packet
Flow
2
Cisco Public
VSG
3
Flow Access Control

Log/Audit
(policy evaluation)
82
Virtual Security Gateway

Performance Acceleration with vPath
VM
VM
VM
VM
VM
VM
VM
VM
VM
VNMC
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Nexus 1000V
vPath
Distributed Virtual Switch
ACL offloaded to
Nexus 1000V
(policy enforcement)
Remaining
packets from flow
BRKDCT-2610
Cisco Public
VSG
Log/Audit
83
VSG: What Problem is Being Solved ?

Control inter-VM traffic
Address new blind spot
Mobility Transparent Enforcement

VM-to-VM traffic
VM-to-VM traffic
VLAN-agnostic Operation
Policy based
App
OS
Administrative Segregation
Server Network Security
BRKDCT-2610
Cisco Public
App
OS
App
OS
App
OS
84
Non-Disruptive Administration
Mitigate Operational errors between teams
Security team defines security policies
Networking team binds port-profile to VSG service profile
Server team Assigns VMs to Nexus 1000V port-profiles
vCenter
Nexus 1KV
Server Admin
BRKDCT-2610
Network Admin
Cisco Public
VNMC
Security Admin
85
VSG Deployment Scenario N1KV
VSG is deployed to protect multiple hosts

Nexus 1000v is deployed with VEM having vPath intelligence
Active
VSG
Standby
VSG
vPath
N1KV VEM
Hypervisor
vPath
N1KV VEM
Hypervisor
vPath
N1KV VEM
Hypervisor
vPath
N1KV VEM
Hypervisor
BRKDCT-2610
VNMC VSM
Cisco Public
86
Securing Virtual Desktops (Use Case)

Persistent virtual
workspace for the doctor
Flexible workspace for

Doctors assistant
Server Zones
Healthcare Portal
Records
Database
Application
Virtual Security Gateway (VSG)

HVD Zones
IT Admin
Maintain compliance
while supporting IT
consumerisation
Assistant
Doctor
Guest
ASA
Network
iT Admin
Guest
Doctor
Cisco AnyConnect
BRKDCT-2610
Cisco Public
87
Summary
Discussed Current Data Centre Challenges
Reviewed solutions to accomplish active / active Layer2
forwarding paths
Reviewed solutions for active / active FHRP
Workload mobility at scale within a Data Centre as well as

across Data Centres
Access layer solutions for 100Mb, 1GbE, 10GbE, Unified I/O
and Storage Integration with a standards based approach
Virtual access layer networking and security benefits to
achieve the dynamic elements of server virtualisation.
BRKDCT-2610
Cisco Public
88
Q&A
BRKDCT-2610
Cisco Public
89
Complete Your Online Session Evaluation

Complete your session evaluation:
Directly from your mobile device by visiting
www.ciscoliveaustralia.com/mobile and login
by entering your badge ID (located on the
front of your badge)
Visit one of the Cisco Live internet stations
located throughout the venue
Open a browser on your own computer to
access the Cisco Live onsite portal
BRKDCT-2610
Cisco Public
90

BRKDCT 2610

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCT 2610

Uploaded by

Copyright:

Available Formats

Next Generation Data Centre Architecture

BRKDCT-2011 - Design and Deployment of Data Centre Interconnects

2011 Cisco and/or its affiliates. All rights reserved.

Next Generation Data Centre Technologies

2011 Cisco and/or its affiliates. All rights reserved.

Data Centre Facilities and Network

2011 Cisco and/or its affiliates. All rights reserved.

What are the implications

Power & Space

Cooling and Airflow

Brownfield DCs are aging fast and are hard to retrofit!

Greendfield DCs are carefully planned, 18-24 months ahead

Racks and Cabinets

Infrastructure choices affect the network architecture

What is happening the next 24 months?

Migration from GE to 10GE attached servers

Adoption of 40GE technologies: switch interconnect and servers

Increase Adoption of Virtualised Technology

Start of migration to non-STP environments: IS-IS and ECMP L2/L3

2011 Cisco and/or its affiliates. All rights reserved.

Data Centre Evolution Path

Increase in 10 Gigabit Ethernet port density

2011 Cisco and/or its affiliates. All rights reserved.

The Evolving Data Centre Architecture

Dedicated service switches

Servers connected to 1G ports at

Architecture is based on optimised

2011 Cisco and/or its affiliates. All rights reserved.

2011 Cisco and/or its affiliates. All rights reserved.

Current STP Deployments

STP blocks redundant uplinks

VLAN based load balancing

Loop Resolution relies on STP

2011 Cisco and/or its affiliates. All rights reserved.

Half of the links in the

2011 Cisco and/or its affiliates. All rights reserved.

All links active and forwarding to

2011 Cisco and/or its affiliates. All rights reserved.

Evolving Data Centre Architecture

Data Centre Row 2

How much capacity does a server need

Where and how do you connect the

2011 Cisco and/or its affiliates. All rights reserved.

Current Challenges in the Virtualised Data Centre

Provisioning of network services for VMs (Port profiles, etc.)

Lack of visibility of VM to VM traffic

Deployment of advanced functionality down to the VMs (ACLs, security, etc.)

Scaling management applications to match growth in deployed switches

Lack of common management tools

Difficulty in segregating server and network management functions

2011 Cisco and/or its affiliates. All rights reserved.

Next Generation Data Centre Technologies

2011 Cisco and/or its affiliates. All rights reserved.

The Evolving Data Centre Architecture

Dedicated service switches

Servers connected to 1G ports at

Architecture is based on optimised

2011 Cisco and/or its affiliates. All rights reserved.

Virtual Port-Channels (vPC)

2011 Cisco and/or its affiliates. All rights reserved.

Virtual Port Channel - vPC

Virtual Port Channel

Provides increased bandwidth