You are on page 1of 96

Case Study in Network Virtualisation

UWA
BRKRST-2068

www.ciscolivevirtual.com
Agenda

Overview of Network
Virtualisation
Concepts and benefits

Mechanisms of Network
Virtualisation
Switching, routing, shared services

Case Study
University of Western Australia (UWA)

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Network Virtualisation
Overview
Concept - Network Virtualisation
One physical network many virtual networks
Department Function B Guests / Partners
A

Virtual Virtual Virtual

Actual Physical Network Infrastructure


BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Network Virtualisation
Groups and services
Resources Internet
are logically separated
Guest / partner access
Departmental separation
Non-Virtualised Regulatory compliance (PCI )
Network Building controls, video surveillance

Virtualised
Closed User Groups
Network Private
Independent policies

Service differentiation
is configured per group / service
Dept A Dept B Partner Guest
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
What does this mean for business ?
Ability to create separate, logical networks and place specific
classes of traffic within the confines of the virtual network
construct can result in:
Elimination of duplicate hardware
Less administrative overhead through
reduction of policy enforcement points.
Ease of integration of two where
overlapping networks may exist.
Reduce carbon footprint
Minimise TCO, Maximise ROI

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Example Use Cases
Education Uni campus divided into administrative and student networks as
well as guest access and departmental isolation.

Retail Network separation required for all point-of-sale (POS) equipment for
Payment Card Industry (PCI) compliance.

Mining and Energy - Isolation of critical infrastructure communications, such as


power generation and transmission data from administrative traffic.

Government Isolation of various hosted government agencies from one


another on a common infrastructure.

Healthcare Networks defined for imaging, clinical staff access, administration,


patient guest access, security services.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Network Virtualisation
Mechanisms
Network Virtualisation
Functional Architecture
Service Access Distribution/Core Services Edge
Branch Campus WAN MAN Campus Data Centre Internet Edge -
Shared Services

GRE MPLS

VRFs

Internet

Authenticate client Transport traffic over Provide access to


isolated Layer 3 partitions services
Functions Authorise client
into a partition (VLAN) Map Layer 3 isolated path Apply policy per partition
to access and services edge
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Layer 2 Virtualisation
VLANs provide the most basic means of isolating network
traffic at Layer 2 in a bridged/broadcast domain.
Require a Layer 3 device to route between those domains.

A Static/Dynamic VLAN assignment

VLAN 10

B VLAN
VLAN 20
TRUNK
VLAN 30

C
Access Switch Router
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Limitations of Layer 2 Networks
Scalability
Broadcasts, MAC flooding
Slow convergence
STP 30-50s, RSTP 2-3 sec
Painful to troubleshoot
(Unless you enjoy hunting for MAC addresses)
High touch provisioning
VLANs defined on every switch OR VTP
Inefficient bandwidth use
STP can only use a single path per VLAN
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Limitations of Layer 2 Networks
Scalability
Broadcasts, MAC flooding
Slow convergence
STP 30-50s, RSTP 2-3 sec
Painful to troubleshoot
(Unless you enjoy hunting for MAC addresses)
High touch provisioning
VLANs defined on every switch OR VTP
Inefficient bandwidth use
STP can only use a single path per VLAN
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Network Architecture Evolution
Large Switched/Minimal Routing Large Routed/Minimal Switching

Internet
Internet

Service
L3 Edge
Core
Core

L3
Dist Dist Si Si Si

L2

Acc Acc L2

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Network Virtualisation at
Layer 3
Layer 3 Virtualisation Control Plane
Router virtualisation
Control plane virtualisation
Data path virtualisation

Control plane uses concept of


Virtual Routing and Forwarding (VRF)

Each VRF has its own routing table


and FIB
Router# show ip route vrf BLUE
Physical or logical interfaces are Routing table: BLUE

assigned to a particular VRF Gateway of last resort is not set


B 51.0.0.0/8 [200/0] via 10.13.13.13, 00:24:19
C 50.0.0.0/8 is directly connected, Ethernet1/3
Interfaces not assigned to a VRF B 11.0.0.0/8 [20/0] via 10.0.0.1, 02:10:22
are associated with the global routing table B 12.0.0.0/8 [200/0] via 10.13.13.13, 00:24:20

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Layer 3 Virtualisation Data Path
Hop-by-Hop
VRF-Lite or EVN (Easy Virtual Network)
802.1q/VNET tag for separation

Multi-Hop
VRF-Lite or EVN + GRE
GRE for separation

Multi-Hop
MPLS-VPN
MPLS Labels for separation

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
MPLS VPN How Does it Work ?
1. Create L2 VLANs
and trunk them to the first L3 device

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
MPLS VPN How Does it Work ?
1. Create L2 VLANs
and trunk them to the first L3 device
2. Define VRFs at the first L3 devices (PE)
and map the L2 VLANs to the proper VRF
PE

PE

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
MPLS VPN How Does it Work ?
1. Create L2 VLANs
and trunk them to the first L3 device
2. Define VRFs at the first L3 devices (PE)
and map the L2 VLANs to the proper VRF
PE
3. Enable MPLS on all Layer 3 interfaces Enable MPLS
in the network

P P

Enable MPLS

PE

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
MPLS VPN How Does it Work ?
1. Create L2 VLANs
and trunk them to the first L3 device
2. Define VRFs at the first L3 devices (PE)
and map the L2 VLANs to the proper VRF
PE
3. Enable MPLS on all Layer 3 interfaces
in the network
4. Enable MP-BGP on the PE devices to P
exchange VPN routes. P
PEs become iBGP neighbors

PE

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
MPLS VPN How Does it Work ?
1. Create L2 VLANs
and trunk them to the first L3 device
2. Define VRFs at the first L3 devices (PE)
and map the L2 VLANs to the proper VRF
PE
3. Enable MPLS on all Layer 3 interfaces
in the network
4. Enable MP-BGP on the PE devices to P
exchange VPN routes. P
PEs become iBGP neighbors
5. VPN traffic is now carried end-to-end
across the network, maintaining logical PE
isolation between the defined groups.
Each frame is double-tagged
(IGP label + VPN label)
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
For Your
MPLS VPN Reference
BGP Scalability: iBGP Neighbor Relationships

iBGP requires a full mesh of neighbors


172.16.5.0/24 172.16.8.0/24

S2 S3
R1S1 R4S8
172.17.6.0/24 172.17.9.0/24

172.18.7.0/24 172.18.10.0/24

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
For Your
MPLS VPN Reference
BGP Scalability: iBGP Neighbor Relationships

iBGP requires a full mesh of neighbors


172.16.5.0/24 172.16.8.0/24
N * (N-1) / 2 = 8 * 7 / 2 = 28
S2 S3
R1S1 R4S8
172.17.6.0/24 172.17.9.0/24

172.18.7.0/24 172.18.10.0/24

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
For Your
MPLS VPN Reference
BGP Scalability: Route Reflectors

Route Reflector Route Reflector

172.16.5.0/24 172.16.8.0/24

S2 S3
R1S1 R4S8
172.17.6.0/24 172.17.9.0/24

172.18.7.0/24 172.18.10.0/24

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
For Your
MPLS VPN Reference
BGP Scalability: Route Reflectors

Route Reflector Route Reflector

172.16.5.0/24 172.16.8.0/24

S2 S3
R1S1 R4S8
172.17.6.0/24 172.17.9.0/24

172.18.7.0/24 172.18.10.0/24

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
For Your
MPLS VPN Reference
BGP Scalability: Route Reflectors

Route Reflector Route Reflector

172.16.5.0/24 172.16.8.0/24

S2 S3
R1S1 R4S8
172.17.6.0/24 172.17.9.0/24

172.18.7.0/24 172.18.10.0/24

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Sharing Services in a Layer 3
Virtualised Environment
Inter-VRF Communication
Communication between VRFs via Fusion Router

Unprotected
Shared
Services !

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Network Virtualisation
Deploying a Centrally Managed Firewall
Single device performs both routing and Shared Services
firewall functionality E-mail
Storage
Web

Non-virtualised
firewall for centralised
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Network Virtualisation
Deploying a Centrally Managed Firewall
Single device performs both routing and Shared Services
firewall functionality E-mail
Storage
Web
Fusion router/firewall establishes routing with
the various VRFs

Non-virtualised
firewall for centralised
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Network Virtualisation
Deploying a Centrally Managed Firewall
Single device performs both routing and Shared Services
firewall functionality E-mail
Storage
Web
Fusion router/firewall establishes routing with
the various VRFs EIGRP, OSPF,
RIPv2, Static,
The fusion router/firewall would typically BGP
advertise only a default route
into the various VRFs Non-virtualised
firewall for centralised
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Network Virtualisation
Deploying a Centrally Managed Firewall
Single device performs both routing and Shared Services
firewall functionality E-mail
Storage
Web
Fusion router/firewall establishes routing with
the various VRFs EIGRP, OSPF,
RIPv2, Static,
The fusion router/firewall would typically BGP
advertise only a default route
into the various VRFs Non-virtualised
firewall for centralised
policy control
Single IGP instance / process, multiple
neighbors
Red Blue Green
Firewall policies centrally managed by IS VPN VPN VPN
department
Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Network Virtualisation
Deploying Virtualised Firewall in Transparent Mode
Firewall contexts in transparent mode act Shared Services
as L2 bridges E-mail
Storage
Web

Virtualised firewall L2 L2 L2
instances for devolved
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Network Virtualisation
Deploying Virtualised Firewall in Transparent Mode
Firewall contexts in transparent mode act Shared Services
as L2 bridges E-mail
Storage
Web
Fusion router establishes routing with the
various VRFs EIGRP, OSPF,
eBGP, RIPv2,
Static

Virtualised firewall L2 L2 L2
instances for devolved
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Network Virtualisation
Deploying Virtualised Firewall in Transparent Mode
Firewall contexts in transparent mode act Shared Services
as L2 bridges E-mail
Storage
Web
Fusion router establishes routing with the
various VRFs EIGRP, OSPF,
eBGP, RIPv2,
The fusion router could typically advertise only Static
a default route into the various VRFs
Virtualised firewall L2 L2 L2
instances for devolved
A dedicated Fusion VRF may be policy control
used in place of an external fusion
router device
Red Blue Green
Each department has their own virtual VPN VPN VPN
firewall, allowing for self-management
Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Network Virtualisation
Deploying Virtualised Firewall in Routed Mode
Firewall contexts in routed mode act as Shared Services
an L3 hop E-mail
Storage
Currently no routing protocol support on Web

firewall deployed in multi-context mode

Virtualised firewall L3 L3 L3
instances for devolved
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Network Virtualisation
Deploying Virtualised Firewall in Routed Mode
Firewall contexts in routed mode act as Shared Services
an L3 hop E-mail
Storage
Currently no routing protocol support on Web

firewall deployed in multi-context mode

The only recommended peering protocol is eBGP/static


eBGP or Static Routing
Virtualised firewall L3 L3 L3
instances for devolved
policy control

Red Blue Green


VPN VPN VPN

Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Network Virtualisation
Deploying Virtualised Firewall in Routed Mode
Firewall contexts in routed mode act as Shared Services
an L3 hop E-mail
Storage
Currently no routing protocol support on Web

firewall deployed in multi-context mode

The only recommended peering protocol is eBGP/static


eBGP or Static Routing
Virtualised firewall L3 L3 L3
The fusion router could typically advertise instances for devolved
only a default route into the various VRFs policy control

A dedicated Fusion VRF may be used in


place of an external fusion router device Red Blue Green
VPN VPN VPN
Each department has their own virtual
firewall, allowing for self-management Campus Core
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Network Virtualisation In Summary
Service Access Distribution/Core Services Edge
Branch Campus WAN MAN Campus Data Centre Internet Edge -
Shared Services

GRE MPLS

VRFs

Internet

Layer 2 VLANs VRF-Lite, EVN Fusion Router/FW


Virtualisation Layer 3 VRF-Lite, EVN VRF Lite + GRE Transparent virtualised
Options MPLS VPN firewall
Routed virtualised
firewall
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Case Study:
University of Western Australia
(UWA)
UWA - General Information
The main UWA Campus is located in
Crawley, 4km west of Perth.
Over 21,000 students, 1400 staff.
UWA Campus is approx 500,000sqm in
extent, covers over 200 buildings in total.
Four additional (smaller) Campuses
located around WA.
Accounts for 70% of the academic research
in Western Australia.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
UWA - Original Network Facts

Information Services team manages 330 routers and switches


34,000 wired Ethernet ports
400 Wireless Access Points
250+ VLANs
Number of networks/subnets = ~500
GbE within Core, 100Mbps to each department
Total Internet bandwidth used of 500Mbps

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
UWA - Original Network Architecture

~ 60 devices

Flat layer 2 with STP

~250 VLANs

Centralised FWs

Only Core and


Distribution is shown.
Where is the Access
layer ?

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
UWA Access Layer Architecture
Access network infrastructure and design wholly owned and
managed by the respective departments.
Each faculty or school assigned one or more VLANs within the
core for segregation -
servers, students, faculty, staff, admin, labs etc.

Many VLANs spanned multiple buildings using flat address


space.
Each department manages its own virtualised firewall.
Access layer will be upgraded separately after the Core and
Distribution networks.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
UWA Customer Experience Goals
Speed
Offer an increased connectivity speed to end hosts of 1-10Gbps and be positioned to take
advantage of speed/bandwidth improvements such as 100 Gbps at the Core Network without
the need for forklift upgrades.

Availability
Improve the resilience and capacity of the network by deploying redundant and diverse access
layer connectivity. In the event of a node or link failure, no perceptible performance impact to
end-users should be apparent.

Functionality
Offer a better quality of experience to staff and students via a traffic management framework to
ensure business-critical applications such as voice and video are prioritised.

Consistency
Improve management of the networking resources across all of campus by standardising on
network platforms and eliminating the need for ongoing maintenance and reconfiguration of
the network to minimise the impact to faculties and users.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
UWA Business Goals
Standardisation
Be able to offer a defined and extensible networking service across its campuses and publish a
baseline service level.

Security
Improve the physical and logical security of the network by centralising firewall management.

Efficiency
Oversee the cost and lifecycle management of the active networking hardware together with a
simplified maintenance regime across all devices.

Scalability
Better manage its scarce IPv4 address allocation and equally introduce IPv6 with commonality
of purpose across all its campuses.

Environmental impact
Deliver an improved environmental footprint by consolidation of networking hardware.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
UWA Original Network Challenges
100Mbps to access layer a bottleneck
Performance Core limited to 1000Mbps insufficient for DC requirements
No differentiated services or QoS

L2 architecture difficult to scale without performance impact


Scalability Flat designs lack structure or hierarchy
Unable to utilise multiple data paths

Slow convergence from STP, non deterministic


Availability Fate-sharing between departments
Limited hardware redundancy

High touch provisioning, no access layer visibility


Manageability Network changes prone to inducing the Butterfly Effect
L2 very time consuming to troubleshoot
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
UWA Case Study:
The Solution
UWA Solution Layer 2 Virtualisation ?
Technically, it would be possible to bridge VLANs
across the core UWA network
This has often been requested historically.
Layer 2 is simple and just works.

However, in practice this creates many difficulties


It does not scale well.
The proliferation of Layer 2 loops in a fully-resilient network design leads to
many blocking / forwarding ports this is considered to be undesired in
UWAs network design.

As a consequence, native bridging of VLANs across the core


was not supported by UWA.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
UWA Solution: Layer 3 Virtualisation
L3 VPNs using VRFs provide for a separate IP
routing table per department or faculty even when
distributed over multiple buildings.

VRFs provide the required


segmentation to provide
a virtual network for the
department or faculty
involved.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
UWA: Network Virtualisation Architecture
UWA required a solution that could scale to handle the large number
of departments and faculties on Campus.
The options for a Network Virtualisation
deployment included
VRF-Lite with GRE Tunnels
VRF-Lite End-to-End, or
MPLS VPNs

For the scale requirements


of UWA, an MPLS VPN
deployment model was chosen.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
UWA Core Physical Architecture
10G Internet Internet 10G ASR1006
Internet
Edge
40G
ASR9010

Data Centre ASA5585


Distribution
100G

Core
ASR9010

40G
Campus
Catalyst 6509
Distribution
Sup720-3C

Campus
Access 10G

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Cisco ASR 9000 At a Glance For Your
Reference

ASR = Advanced Services Router


6 slot and 10 slot version currently shipping
Optimised for dense 10GE & 100GE
Up to 400Gbps (FDX) of bandwidth per slot
Chassis capacity for each of UWAs four 9010s
is 6.4Tbps
Based on IOS-XR for non-stop availability
Widely deployed throughout SPs globally

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
UWA: MPLS VPN Network Design
The first step was determining where to place the P (Provider)
and PE (Provider Edge) routers.

CORE

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
UWA: MPLS VPN Network Design
The first step was determining where to place the P (Provider)
and PE (Provider Edge) routers.

The Core P
CORE P
ASR9010s are the
P routers.

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
UWA: MPLS VPN Network Design
The first step was determining where to place the P (Provider)
and PE (Provider Edge) routers.
The Distribution
PE PE
ASR9010s are DC
PE routers.
The Core P
CORE P
ASR9010s are the
P routers.

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
UWA: MPLS VPN Network Design
The first step was determining where to place the P (Provider)
and PE (Provider Edge) routers.
The Distribution
PE PE
ASR9010s are DC
PE routers.
The Core P
CORE P
ASR9010s are the
PE PE
P routers.

The Distribution
6509s are PE PE PE PE PE PE PE PE

PE routers Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
UWA: mBGP Routing Design
MPLS VPN uses Multi-Protocol BGP (MP-BGP) for
exchanging MPLS tags (for creating isolated routing
domains).
PE PE

P
CORE P

PE PE

PE PE PE PE PE PE PE PE

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
UWA: mBGP Routing Design
MPLS VPN uses Multi-Protocol BGP (MP-BGP) for
exchanging MPLS tags (for creating isolated routing
domains).
PE PE
The first step was
selecting the
Route Reflectors
CORE
for MP-BGP. P P

PE PE

PE PE PE PE PE PE PE PE

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
UWA: mBGP Routing Design
MPLS VPN uses Multi-Protocol BGP (MP-BGP) for
exchanging MPLS tags (for creating isolated routing
domains).
PE PE
The first step was RR RR

selecting the
Route Reflectors
CORE
for MP-BGP. P P

PE PE
UWA selected the
Data Centre Distribution
routers as the RRs
PE PE PE PE PE PE PE PE
(separate RRs could have
been deployed) Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
UWA: mBGP Routing Design
After determining the BGP route reflector locations, it was
necessary to set up the appropriate routing.

PE PE
RR RR

P
CORE P

PE PE

PE PE PE PE PE PE PE PE

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
UWA: mBGP Routing Design
After determining the BGP route reflector locations, it was
necessary to set up the appropriate routing.

All of the Distribution Point 6509s


PE PE
peer (MP-BGP) with the two ASR RR RR

9010 Route Reflectors for


MPLS VPN (VRF) route exchange.
P
CORE P

PE PE

PE PE PE PE PE PE PE PE

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
UWA: mBGP Routing Design
After determining the BGP route reflector locations, it was
necessary to set up the appropriate routing.

All of the Distribution Point 6509s


PE PE
peer (MP-BGP) with the two ASR RR RR

9010 Route Reflectors for


MPLS VPN (VRF) route exchange.
P
CORE P
All BGP peering is done
between loopbacks. PE PE

PE PE PE PE PE PE PE PE

Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
UWA: mBGP Routing Design
After determining the BGP route reflector locations, it was
necessary to set up the appropriate routing.

All of the Distribution Point 6509s


PE PE
peer (MP-BGP) with the two ASR RR RR

9010 Route Reflectors for


MPLS VPN (VRF) route exchange.
P
CORE P
All BGP peering is done
between loopbacks. PE PE

The Core routers are not


aware of VRFs and do not run
MP-BGP OSPF only, with PE PE PE PE PE PE PE PE

MPLS tagging. This makes the


Core zero-touch. Distribution

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
UWA: Distribution Layer Architecture
Next step was providing for layer 3 termination of the access
VLANs, mapping into VRFs, and first hop redundancy.
Traditional L2
Each pair of Distribution Point 6509s is Access
configured as a Virtual System Switching Redundancy

(VSS) pair.

VSS appears as a single logical device.

Multichassis Etherchannel (MEC) is used


for access layer redundancy.

No need for HSRP or Spanning Tree.

Very fast (100ms) failover.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
UWA: Distribution Layer Architecture
Next step was providing for layer 3 termination of the access
VLANs, mapping into VRFs, and first hop redundancy.
Traditional L2
Each pair of Distribution Point 6509s is Access
configured as a Virtual System Switching Redundancy

(VSS) pair.

VSS appears as a single logical device.

Multichassis Etherchannel (MEC) is used


for access layer redundancy.

No need for HSRP or Spanning Tree.

Very fast (100ms) failover.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
UWA: Distribution Layer Architecture
Next step was providing for layer 3 termination of the access
VLANs, mapping into VRFs, and first hop redundancy.
Each pair of Distribution Point 6509s is
configured as a Virtual System Switching
(VSS) pair.

VSS appears as a single logical device.

Multichassis Etherchannel (MEC) is used


for access layer redundancy.

No need for HSRP or Spanning Tree.

Very fast (100ms) failover.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
UWA: VRF and VLAN Definition
VLAN and VRF definitions at the PE routers
Note VRFs are only defined on the PE routers where the VRFs need to terminate
not all VRFs are defined on all PE routers
vlan 201
name VLAN-201-IS-MGT
VLAN definition Layer 2

ip vrf IS-MGT
rd 65000:200
route-target export 65000:200 VRF definition Layer 3
route-target import 65000:200

The Route Distinguisher is a 64-bit Routes are imported and exported


value (unique per VRF). When added within the VRF using route-targets
to the 32-bit IP address, this forms a to populate the VRFs IP routing table.
unique 96-bit VPNv4 address.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
UWA: Mapping L2 and L3 Virtualisation
SVI definition on the PE router, showing SVIs being assigned into a VRF.
VRF members can also be L3 subinterfaces, tunnels or physical interfaces.

DP1-2# DP1-2# show run int Vlan 201


! !
vlan 201 interface Vlan201
name VLAN-201-IS-MGT description VLAN-201-IS-MGT
ip vrf forwarding IS-MGT
ip vrf IS-MGT ip address 172.16.10.253 255.255.255.0
rd 65000:200
route-target export 65000:200
route-target import 65000:200

Step 1. Define VLAN and VRF Step 2. Create logical interface (SVI in this ex.)
Step 3. Assign interface to VRF
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
UWA: NV Route Propagation
Connected routes and static routes within the VRF are redistributed into
MP-BGP making these routes available on other PEs that also host this
VRF

interface Vlan201 router bgp 65000


ip vrf forwarding IS-MGT !
ip address 172.16.2.254 255.255.255.0 address-family ipv4 vrf IS-MGT
redistribute connected
interface Vlan202 redistribute static
ip vrf forwarding IS-MGT no synchronization
ip address 172.16.5.254 255.255.255.0 network 0.0.0.0
exit-address-family
interface Vlan203
ip vrf forwarding IS-MGT
ip address 172.16.10.254 255.255.255.0 Redistribution of routes to other
PEs (via MP-BGP)

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
UWA: Virtualised Routing Table
Now that we have done all of our VLAN and VRF
definitions, SVI configuration, and route redistribution

DP-2# show ip route vrf IS-MGT

Routing Table: IS-MGT


Codes: C - connected, S - static, R - RIP, M - mobile, B BGP
. . .
Gateway of last resort is 172.16.2.249 to network 0.0.0.0

172.16.0.0/16 is variably subnetted, 3 subnets, 3 masks


B 172.16.139.0/24 [200/0] via 192.168.1.3, 5w3d mBGP routes
B 172.16.155.128/26 [200/0] via 192.168.1.5, 5w3d
C 172.16.10.0/24 is directly connected, Vlan203
S* 0.0.0.0/0 [1/0] via 172.16.2.249

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
UWA: Inter-VRF Communication
Communication between VRFs is accomplished via a fusion device.
In UWAs network, this is a pair of ASA 5585 FW/IDS devices.
Virtualisation of the firewall is not required (centralised management
policy).
2 x 10G 2 x 10G
ASA 5585 ASA 5585
2 x 10G 2 x 10G

Data Centre Data Centre


Distribution ASR9010 Distribution ASR9010

Core Core
ASR9010 ASR9010

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
UWA: Inter-VRF Communication
Redistribute BGP
into EIGRP, and EIGRP Process
EIGRP into BGP per VPN

VPN A Subinterface 1 Subinterface 1


EIGRP 1

VPN B Subinterface 2 Subinterface 2 Single EIGRP Inter-VPN


mBGP Routing
EIGRP 2 Process

VPN C Subinterface 3 Subinterface 3


EIGRP 3
20Gbps
LAG

Data Centre
Data Centre
Firewall
Distribution
Router

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
UWA: Access to Shared Services
Access to external shared services is via the data centre
firewalls and perimeter firewalls for layer security.
Internet Border Routers/
Perimeter Firewall

Internet Internet

ASR 1006 ASR 1006

2 x 10G 2 x 10G
ASA 5585
2 x 10G 2 x 10G

Data Centre Core Data Centre


Distribution ASR9010 Distribution ASR9010

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
UWA: Access to Shared Services
Global Routing
Table

Internet
via ASR1006 BGP 20Gbps
LAG

Internal EIGRP
routes redistributed
into BGP

Subinterface 1 Subinterface 1
VPN A

Subinterface 2 Subinterface 2 Single EIGRP


mBGP VPN B Process
20Gbps
LAG

Subinterface 3 Subinterface 3 Data Centre


Data Centre VPN C Firewall
Distribution
Router

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
UWA: Shared Services: Multicast
Multicast is used at UWA to deploy desktop
software updates from a centralised repository.

An Extranet mVPN is the central source mVPN


where the software deployment servers reside.

Each reciever VPN will have direct


multicast access to the source VPN.

As the mVPN content is trusted, faculty


VRF access is direct, bypassing the
firewall.

Care must be taken to avoid creating


backdoor reachability between VRFs via the
extranet VRF.

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
UWA: Traffic Flow
Internet Internet

Intra-VPN traffic
flows via the shortest
IGP path
Intra-VPN
Traffic Flow

Architecture
Scalability

VPN B VPN A VPN B


BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
UWA: Traffic Flow
Internet Internet

Inter-VPN traffic always Inter-VPN traffic


flows via the internal Flow
firewalls Architecture
Scalability

VPN B VPN A VPN B


BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
UWA: Traffic Flow
Internet Internet

Architecture
Scalability

External traffic always


flows via both the
internal and external
firewalls
VPN B VPN A VPN B
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
UWA: Traffic Flow
Internet Internet

Intra-VPN traffic
flows via the shortest
IGP path
Intra-VPN
Traffic Flow
Inter-VPN traffic always Inter-VPN traffic
flows via the internal Flow
firewalls Architecture
External Traffic
Flow
Scalability

External traffic always


flows via both the
internal and external
firewalls
VPN B VPN A VPN B
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
UWA: Network Virtualisation Maintenance
Network maintenance and troubleshooting tasks
are readily adapted into a multi-VRF environment
ping within a VRF traceroute within a VRF

DP-2# ping vrf IS-MGT 172.16.155.188 DP-2# traceroute vrf IS-MGT


172.16.155.188
Sending 5, 100-byte ICMP Echos to
172.16.155.188, Type escape sequence to abort.
timeout is 2 seconds: Tracing the route to 172.16.155.188

!!!!! 1 10.1.1.1 [MPLS: Labels 165/27 Exp 0]


4 msec 0 msec 0 msec
Success rate is 100 percent (5/5),
round-trip min/avg/max = 1/1/1 ms 2 172.16.155.188
0 msec 0 msec 0 msec
DP-2# DP-2#
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
UWA: Layer 2 Connectivity Requirements
By default, there is no layer 2 Core
connectivity between distribution
point PEs.
Routed DP
Some departments have had DP

to re-address infrastructure
where legacy flat networks L3 SVI SVI
cross a routed interface. L2
Specific technologies may
require L2 connectivity
(e.g. vMotion) or are difficult
to readdress (e.g. building
VPN A VPN A VPN A
management systems) Building 1 Building 1 Building 1

192.168.32.0/22 192.168.36.0/23
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
UWA: Layer 2 Connectivity Options
MPLS core allows layer 2 VPN services to be provisioned.
Ethernet over MPLS (EoMPLS)
Also called pseudowire, E-LINE, VPWS
Point-to-point only
Port-based or VLAN-based

VPLS P2MP or Full Mesh


UWA has deployed EoMPLS
as an interim solution for:
L2 connectivity between DCs during server migrations.
L2 connectivity for departments where there is a delay in re-addressing end
devices for connectivity to layer 3 core.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
UWA Case Study:
Next Steps
UWA: Next Steps - IPv6
UWA is examining the use of IPv6 on Campus
Small deployments underway, with a larger Campus-wide deployment in
planning.
Catalyst 6500 and ASR9010
devices offer support for 6VPE
capability.
No config required on Core
Building IPv6-Ready Campus
Networks:
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/solution_overview_c22-531339.html

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
UWA Next Steps: Access Layer
Option 1 Layer 2 Access
Current design of legacy access DP

network
Pros Distribution
SVI
L3
Simple L2 devices in access Access L2
Cons 1/10/40Gbps VLAN
(Optional LAG) Trunk
No dual homing to separate Dist. Points
Limited scalability (DP sees all MACs)
Complexity on DP devices RSTP etc
RSTP
DP exposed to L2 events within access
L2 does not promote hierarchical design Faculty A Network

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
UWA Next Steps: Access Layer
Option 2 Layer 3 Access
L3 subinterfaces between DP DP DP

and L3 access switch for each VRF


or use Cisco Easy Virtual Network
(EVN)
Pros Routed P2P
Access can dual home to different DPs subinterfaces
(/30 or /31s)
Complexity removed from distribution layer
Isolate L2 complexity to within access network L3
Scalable to nth degree L2 SVI

Cons
Faculty A Network
L3 capable infrastructure required in access
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
UWA Next Steps: Access Layer
Option 3 MPLS Access
Extend MPLS down to the access layer DP DP

Pros (in addition to L3 access)


Distribution layer is now zero touch
Isolate all configuration complexity to
within access network
No BGP routing on distribution not P2P MPLS link P2P MPLS link
globally reachable
EoMPLS/VPLS services accessible from MPLS
access layer L2/L3 SVI

Cons
MPLS capable infrastructure required in access Faculty A Network

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Summary
UWA: Facultys Network Perspective

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
UWA: Universitys Network Perspective
Internet Internet

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
UWA: A Foundation for Education

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Additional Resources
When designing, planning, and
building UWAs Network Virtualisation
deployment, Ciscos NV Design
Guides were used extensively

NV Access Control Design Guide


NV Path Isolation Design Guide
NV Services Edge Design Guide

www.cisco.com/go/designzone

BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Q&A
Complete Your Online Session
Evaluation
Complete your session evaluation:
Directly from your mobile device by visiting
www.ciscoliveaustralia.com/mobile and login
by entering your username and password

Visit one of the Cisco Live internet


stations located throughout the venue

Open a browser on your own computer


to access the Cisco Live onsite portal Dont forget to activate your Cisco Live
Virtual account for access to all session
materials, communities, and on-demand and
live activities throughout the year. Activate your
account at any internet station or visit
www.ciscolivevirtual.com.
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
BRKRST-2068 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 96

You might also like