BRKACI-2001 - Integration and Interoperation of Existing Nexus Networks Into An ACI Architecture (2015 Melbourne)

Integration and Interoperation of
Existing Nexus Networks into an ACI

Architecture
BRKACI-2001
Mike Herbert Principal Engineer INSBU
#clmel
Introducing: Application Centric Infrastructure (ACI)

Apps + Infrastructure
Physical + Virtual + Containers
Open + Secure
On-Premises + Cloud
Application Oriented Policy = Operational Simplicity

BRKACI-2001
2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Why Networks are Complex

Overloaded Network Constructs
ACI directly maps the application
connectivity requirements onto the
network and services fabric
Redirect and Load Balance

Connectivity
IP Address, VLAN, VRF
Control & Audit Connectivity

Application Requirements
(Security Firewall, ACL, )
IP Addressing
IP Address,
VLAN, VRF
Enable Connectivity
(The Network)
BRKACI-2001
Cisco Public
Application Requirements
Dynamic provisioning of
connectivity explicitly defined for
the application
Application Specific Connectivity
Why Network Provisioning is Slow

Application Language Barriers
Developers
Infrastructure Teams
Application
Tiers
Provider /
Consumer
Relationships
VLANs
Subnets
Protocols
Ports
Developer and infrastructure teams must translate between disparate languages.
BRKACI-2001
Cisco Public
Centralised Policy and Distributed Enforcement

WEB
F/W
ADC
APP
ADC
DB
SLA
QoS
CONNECTIVIT
Y POLICY
Security
SECURITY
POLICIES
Load
QOS
Balancing
APPLICATION
L4..7
SERVICES
STORAGE
AND
COMPUTE
APP PROFILE
HYPERVISOR
BRKACI-2001
HYPERVISOR
Cisco Public
HYPERVISOR
Two Big Questions

Is ACI a Closed System?
Do I need to replace all of my existing

infrastructure to begin leveraging ACI?
ABSOLUTELY NOT !!!

Lets see WHY and HOW
BRKACI-2001
Cisco Public
Things we would like to understand how to do

Extend ACI to
WAN/DCI
AVS
vSwitch
Let me just run

my network (but
fix my Flooding,
Mobility,
Configuration,
Troubleshooting
challenges)
BRKACI-2001
AVS
vSwitch
Extend ACI to local

hypervisors
Interconnect to existing
DC Networks
Cisco Public
Extend ACI to to existing Nexus

installations via a full ACI VXLAN
Switching Enabled Hypervisor and
remote ACI Physical Leaf
ACI Policy Based Forwarding via an

Integrated Overlay
ACI Fabric An IP network with an Integrated Overlay

Virtual and Physical
APIC
VTEP
VXLAN
IP
VTEP
BRKACI-2001
IP Transport
Payload
VTEP
VTEP
VTEP
vSwitc
VTEP
h
vSwitc
VTEP
h
VTEP
VTEP
Ciscos ACI solution leverages an integrated VXLAN based overlay
IP Network for Transport
VXLAN based tunnel end points (VTEP)
VTEP discovery via infrastructure routing
Directory (Mapping) service for EID (host MAC and IP address) to VTEP lookup
Cisco Public
ACI Fabric Integrated Overlay

Connecting and Extending the Overlay
Multi-Fabric Topologies
VXLAN Based
Fabric
VTEP
VXLAN Enabled
Hypervisor
VTEP
VTEP
VXLAN Enabled
Hypervisor
Service Interconnect to
ASR9K/N7K WAN/DCI
Interworking with ACI
VTEP
VXLAN enabled Hypervisor (FCS)
VXLAN Hardware VTEP (Nexus 9000 standalone, Nexus 3100/7000-F3, ASR9K, )
MP-BGP EVPN based control plane for external VTEP connectivity (post FCS)
BRKACI-2001
Cisco Public

Decoupled Identity, Location & Policy
All Tenant traffic within the Fabric is

tagged with an ACI VXLAN (VXLAN)
header which identifies the policy
attributes of the application end point
within the fabric
Policy attributes are carried by every packet
Policy Group (source group)
Forwarding Group (Tenant, VRF,

Bridge Domain)
Load Balancing Policy
VTEP
Telemetry Policy
AVS
ACI VXLAN (VXLAN) header identifies the

attributes of the application end point within
the fabric
At the ingress port the Fabric translates

an external identifier which can be used
to distinguish different application end
points via the internal VXLAN tagging
format
BRKACI-2001
VTEP
Flags
VNID
MAC
IP
Payload
VTEP
External Identifiers are localised to

specific iLeaf or iLeaf ports (unless
external requirements for
consistency, e.g. downstream
networks)
Cisco Public
SRC
Group
Eth
MAC
Eth
IP
Payload
Payload
802.1Q
IP
Payload
Outer
IP
NVGR
E
IP
Payload
Outer
IP
VXLAN
Eth
IP
Payload
ACI leverages VXLAN

RFC Draft Next IETF Meeting
BRKACI-2001
Cisco Public
Overlay Elements Host Forwarding
IP Forwarding:
MAC Forwarding:
Forwarded using DIPi

address, HW learning of IP
address
Forwarded using DMAC

address, HW learning of
MAC address
10.1.3.11
10.1.3.35
10.6.3.2
10.6.3.17
Forward based on destination IP Address for intra and inter subnet (Default Mode)
Bridge semantics are preserved for intra subnet traffic (no TTL decrement, no MAC
header rewrite, etc.)
Non-IP packets will be forwarded using MAC address. Fabric will learn MACs for non-IP
packets, IP address learning for all other packets
Route if MAC is router-mac, otherwise bridge (standard L2/L3 behaviour)
BRKACI-2001
Cisco Public
Location Independent Forwarding

Layer 2 and Layer 3
10.1.1.10
10.1.3.1
1
10.6.3.2
10.1.3.35
10.1.1.10
Distributed Default Gateway
10.1.3.1
10.6.3.2
1
10.1.3.35
Directed ARP Forwarding
ACI Fabric supports full layer 2 and layer 3 forwarding semantics, no changes required to applications or end point IP
stacks
ACI Fabric provides optimal forwarding for layer 2 and layer 3
Fabric provides a pervasive SVI which allows for a distributed default gateway
Layer 2 and layer 3 traffic is directly forwarded to destination end point
IP ARP/GARP packets are forwarded directly to target end point address contained within ARP/GARP header
(elimination of flooding)
BRKACI-2001
Cisco Public
Where is my default Gateway?
10.1.3.1
10.1.3.1
1
10.6.3.2
10.1.3.35
10.6.3.1
10.1.1.10
Pervasive SVI
10.1.3.1
1
10.6.3.2
10.6.3.2
10.1.3.35
External Default Gateway
Default Gateway can reside internal or external to the Fabric

Pervasive SVI provides a distributed default gateway (anycast gateway)
Subnet default gateway addresses are programmed in all Leaves with end points present for the specific Tenant
IP subnet
Layer 2 and layer 3 traffic is directly forwarded to destination end point
External Gateway is used when Fabric is configured to provide layer 2 transport only for a specific Tenant
BRKACI-2001
Cisco Public
Overlay Elements - Mapping/Directory Proxy

Inline Hardware Mapping DB - 1,000,000+ hosts
Proxy
Global Station Table

contains a local cache
of the fabric endpoints
10.1.3.35
Leaf 3
Proxy A
10.1.3.11
Port 9
BRKACI-2001
Proxy
Proxy
Proxy Station Table contains

addresses of all hosts attached
to the fabric
10.1.3.11
Local Station Table

contains addresses of
all hosts attached
directly to the iLeaf
Proxy
10.1.3.35 Leaf 3
10.1.3.11 Leaf 1
fe80::8e5e Leaf 4
fe80::5b1a Leaf 6
10.1.3.35
fe80::462a:60ff:fef7:8e5e
fe80::62c5:47ff:fe0a:5b1a
The Forwarding Table on the Leaf Switch is divided between local (directly attached) and
global entries
The Leaf global table is a cached portion of the full global table
If an endpoint is not found in the local cache the packet is forwarded to the default
forwarding table in the spine switches (1,000,000+ entries in the spine forwarding table)
Cisco Public
Endpoint Repository (Proxy DataBase)
BRKACI-2001
Cisco Public
Proxy Database (Oracle)
You still have full access to all forwarding, adjacency, ..., information via
CLI and debug commands when you want them
Spine-1# show coop internal info global
BRKACI-2001
Spine-1# show coop internal event-history oracle-adj <IP>
Cisco Public
ACI Endpoint Tracker Application

Tracks all attachment, detachment,
movement of Endpoints in ACI fabric
Stores activity in open source MySQL
Database, allowing query capabilities
Provides foundation for visualisation and
query tools
Some questions that could be solved:

What are all the Endpoints on network?
Where is a specific Endpoint?
What was connected last Thursday
between 3:30am and 4:00am?
What is the history of a given Endpoint?
BRKACI-2001
Cisco Public
GitHub a resource for ACI scripts and tools

ACI Toolkit:
http://datacenter.github.io/acitoolkit/
https://github.com/datacenter/acitoolkit
ACI Diagram
https://github.com/cgascoig/aci-diagram
ACI Endpoint Tracker

http://datacenter.github.io/acitoolkit/docsb
uild/html/endpointtracker.html
BRKACI-2001
Cisco Public
21
ACI
A Policy Based IP Network
IP Network & Integrated
VXLAN
APIC - Policy Controller &

Distributed Management
Information Tree (DMIT)
Proxy (Directory)
Services
VTEP
Physical and Virtual VTEPs

(Policy & Forwarding Edge
Nodes)
VTEP
VXLAN
IP
Payload
VTEP
VTEP
VTEP
VTEP
AVS
AVS
WAN/DCI
Services
Physical and Virtual Endpoints

(Servers) & VMM (Hypervisor vSwitch)
BRKACI-2001
Cisco Public
Physical and Virtual L4-7

Service Nodes
Extending the Network
Transitions will and need to occur independently

Operations
Evolution
Policy Zone A
Policy
Evolution
Policy Zone B
Policy Zone C
vPC
Component
Evolution
Application Container
vPC
vPC
BRKACI-2001
Cisco Public
App
App
App Element
App
App Element
App
Element
App Element
App
Element
App Element
App
Element
App Element
App
Element
App Element
Element
App Element
Element
Element
Extending ACI in to Current Data Centres

Three Stages
Interconnect existing network PODs with new ACI PODs via standard
Layer 2 extensions (VLAN or VXLAN) or via standard Layer 3 routing
(OSPF, BGP)
Leverage an ACI policy/services block attached to any existing Nexus or

Catalyst network to provide L4-L7 services and policy automation for
existing virtual and physical servers
Extend the ACI forwarding and distributed policy capabilities to virtual

and physical leaf switches connected to an existing Nexus/Catalyst IP
based network
BRKACI-2001
Cisco Public

Interconnect Existing to New ACI POD
Layer 2 and Layer 3 interoperation between ACI Fabric and Existing Data Centre builds
Layer 3 interconnect via standard routing interfaces,
OSPF, Static, iBGP (FCS)
MP-BGP, EIGRP, ISIS (Post FCS_
Layer 2 interconnect via standard STP or via VXLAN overlays

Backbone
vPC
Interconnect at
Layer 3
vPC
vPC
BRKACI-2001
Extend Layer 2 VLANs

where required
Cisco Public
vSwitch
AVS
Hyper-V
Connecting/Extending ACI via Layer 2

Layer 2
Layer 2
Layer 2
Lets Look at
the Links
Extend L2 domain beyond ACI fabric - 2 options

1. Manually assign a port to a VLAN which in turn mapped to an EPG. This extend EPG beyond ACI fabric
(EPG == VLAN)
2. Create a L2 connection to outside network. Extend bridge domain beyond ACI fabric. Allow contract
between EPG inside ACI and EPG outside of ACI
TECACI-2009
Cisco Public

Data Path - Encapsulation Normalisation
IP Fabric Using
VXLAN Tagging
Normalised
Encapsulation
Any to Any
VTEP
Localised
Encapsulation
VXLAN
VNID = 5789
802.1Q
VLAN 50
VXLAN
VNID = 11348
NVGRE
VSID = 7456
All traffic within the ACI Fabric is encapsulated with an extended VXLAN header
External VLAN, VXLAN, NVGRE tags are mapped at ingress to an internal VXLAN tag
Forwarding is not limited to, nor constrained within, the encapsulation type or
encapsulation overlay network
External identifies are localised to the Leaf or Leaf port, allowing re-use and/or translation
if required
BRKACI-2001
Cisco Public
28
VXLAN
IP
Payload
Eth
MAC
Payload
Eth
IP
Payload
802.1Q
IP
Payload
Outer
IP
NVGRE
IP
Payload
Outer
IP
VXLAN
Eth
IP
Payload
Normalisation of Ingress
Encapsulation
Extend the EPG

Option 1
Layer 2
VLAN 30
VLAN 20
100.1.1.3
BD
Existing
App
100.1.1.99 100.1.1.7
EPG
100.1.1.5
100.1.1.3
VLANs are localised to the leaf nodes
The same subnet, bridge domain, EPG can be configured as a different VLAN on each leaf
switch
In 1HCY15 VLANs will be port local
TECACI-2009
Cisco Public
Extend the EPG

Option 1
VLAN 10
VLAN 10
VLAN 10
VLAN 30
VLAN 10
Layer 2
VLAN 20
EPG
100.1.1.3
TECACI-2009
100.1.1.5
100.1.1.99 100.1.1.7
100.1.1.3
BD
Existing
App
Single Policy Group (one extended EPG)
Leverage vPC for interconnect (diagram shows a single port-channel which is an option)
BPDU should be enabled on the interconnect ports on the vPC domain

Cisco Public
Assign Port to an EPG
With VMM integration, port is assigned to EPG by

APIC dynamically.
Use Static Binding under EPG to assign

port to EPG
In all other cases, such as connecting to switch,

router, bare metal, port need to be assigned to
EPG manually or use API
The example assigns traffic received on port

eth1/32 with vlan tagging 100 to EPG VLAN
100
TECACI-2009
Cisco Public
Assign Port to EPG

VLAN Tagging Mode
Tagged. Trunk mode

Untagged. Access mode. Port can only be in one
EPG
802.1P Tag. Native VLAN.
No Tagged and Untagged(for different port) config
for same EPG with current software
TECACI-2009
Cisco Public
Assign port eth1/1 with VLAN 100 tagged mode

and port eth1/2 with VLAN 100 untagged mode to
EPG WEB is not supported
Use 802.1P Tag. Port eth1/1 vlan 100 tagged,
eth1/2 vlan 100 902.1P Tag
VLAN to EPG mapping is switch wide significant
Extend the Bridge Domain

Option 2
VLAN 10
Layer 2
VLAN 10
100.1.1.3
VLAN 10
VLAN 10
EPG
Outside
100.1.1.99 100.1.1.7
VLAN 30
VLAN 20
EPG
Inside
100.1.1.5
BD
Existing
App
100.1.1.3
External EPG (policy between the L2 outside EPG and internal EPG)
Leverage vPC for interconnect (diagram shows a single port-channel which is an option)
BPDU should be enabled on the interconnect ports on the vPC domain
L2 outside forces the same external VLAN << fewer operational errors
TECACI-2009
Cisco Public
L2 Outside Connection Configuration Example
Step 1. Create L2 Outside

connection.
Associate with BD.
Specify VLAN ID to connect to

outside L2 network
External Bridge Domain is a way

to specify the VLAN pool for
outside connection.
It is NOT a Bridge Domain.
TECACI-2009
Cisco Public
Step 2. Specify leaf node

and interface providing
L2 outside connection
TECACI-2009
Cisco Public
Step 3. Create external EPG

under L2 outside connection
Step 4. Create contract

between external EPG and
internal EPG
TECACI-2009
Cisco Public
ACI Interaction with STP
No STP running within ACI fabric
BPDU frames are flooded between

ports configured to be members of the
same external L2 Outside (EPG)
No Explicit Configuration required
Hardware forwarding, no interaction
with CPU on leaf or spine switches
for standard BPDU frames
Protects CPU against any L2 flood
that is occurring externally
External switches break any potential

loop upon receiving the flooded BPDU
frame fabric
BPDU filter and BPDU guard can be

enabled with interface policy
TECACI-2009
Cisco Public
APIC
Same L2 Outside
EPG
(e.g. VLAN 10)
STP Root
Switch
ACI Fabric Loopback Protection
Multiple Protection Mechanisms against

external loops
LLDP detects direct loopback cables

between any two switches in the same
fabric
Mis-Cabling Protocol (MCP) is a new link

level loopback packet that detects an
external L2 forwarding loop
MCP frame sent on all VLANs on all Ports

If any switch detects MCP packet arriving on
a port that originated from the same fabric the
port is err-disabled
External devices can leverage

STP/BPDU
MAC/IP move detection and learning

throttling and err-disable
TECACI-2009
Cisco Public
APIC
LLDP Loop
Detection
MCP Loop
Detection
(supported with
11.1 release)
STP Loop
Detection
Managing Flooding Within the BD

VLAN 10
Layer 2
VLAN 10
100.1.1.3
VLAN 10
VLAN 10
EPG
Outside
100.1.1.99 100.1.1.7
VLAN 20
VLAN 30
EPG
App 1
100.1.1.5
BD
Multi
EPG
EPG
App 2
100.1.1.3
In a classical network traffic is flooded with the Bridge Domain (within the VLAN)
You have more control in an ACI Fabric but need to understand what behaviour you want
TECACI-2009
Cisco Public
Managing Flooding Within the Fabric

ARP Unicast
ARP
Firewall Configured as
the Default Gateway
ARP Flooding Disabled
(Default)
TECACI-2009
Cisco Public
Disable ARP Flooding ARP/GARP is

forwarded as a unicast packet within the
fabric based on the host forwarding DB
On egress the ARP/GARP is forwarded as a

flooded frame (supports hosts reachable via
downstream L2 switches)

ARP Flooding
ARP
Firewall Configured as
the Default Gateway
ARP Flooding Enabled
TECACI-2009
Cisco Public
Enabling ARP Flooding ARP/GARP is

flooded within the BD
Commonly used when the default GW is

external to the Fabric

Unknown Unicast Proxy Lookup
HW Proxy
Lookup
Proxy
Unknown
Unicast
Unknown Unicast
Lookup via Proxy
TECACI-2009
Cisco Public
Hosts (MAC, v4, v6) that are not known by a

specific ingress leaf switch are forwarded to
one of the proxies for lookup and inline
rewrite of VTEP address
If the host is not known by any leaf in the

fabric it will be dropped at the proxy (allows
honeypot for scanning attacks)

Unknown Unicast Flooding
Unknown
Unicast
Unknown
Unicast
Flooded
Unknown Unicast
Flooded
TECACI-2009
Cisco Public
Hosts (MAC, v4, v6) that are not known by a

specific ingress leaf switch are flooded to all
ports within the bridge domain
Silent hosts can be installed as static entries

in the proxy (flooding not required for silent
hosts)

Unknown Multicast Mode 1 (Flood)
Unknown
Multicast
Unknown Multicast
Flooded
TECACI-2009
Cisco Public
Unknown Multicast traffic is flooded locally to

all ports in the BD on the same leaf the
source server is attached to
Unknown Multicast traffic is flooded to all

ports in the BD on leaf nodes with a multicast
router port
Unknown Multicast Mode 2 (OMF or Optimised Flood)
Unknown
Multicast
Unknown Multicast
Optimised Flooding
TECACI-2009
Cisco Public
Unknown Multicast traffic is only flooded to

multicast router ports in this mode

Scoping Broadcasts to a micro segment
EPG
B
EPG
A
100.1.1.3
100.1.1.99
100.1.1.72
100.1.1.7
EPG
C
100.1.1.5
100.1.1.3
Traffic Type
11.0(x) Behaviour
11.1(x) Behaviour
ARP
Flood or Unicast
Flood or Unicast
Unknown Unicast
Flood or Leverage Proxy Lookup
Flood or Leverage Proxy Lookup
Unknown IP Multicast
Flood or OMF
Flood or OMF
L2 MCAST, BCAST, Link Local
Flood
Flood within the BD, Flood within the EPG,

Disable Flooding within the BD/EPG
TECACI-2009
Cisco Public
Multi Destination Flooding (Supported with 11.1(x) Q2CY15)
Link Level
BCAST
100.1.1.4
100.1.1.3
EPG A EPG B
Manage
Flooding within
the BD
Cisco Public
100.1.1.7
100.1.1.52
EPG A
EPG B
100.1.1.72 100.1.1.5
EPG B
TECACI-2009
100.1.1.99
Link Level Traffic is either
Contained within the EPG
Contained within the Bridge Domain
Dropped
Security Segmentation for Link Level Traffic

Flooding scoped to the EPG
EPG
B
EPG
A
100.1.1.3
100.1.1.72
100.1.1.7
100.1.1.5
100.1.1.3
Link Local, BCAST & L2 Multicast traffic can be managed on a micro-segment basis
As an example:
TECACI-2009
100.1.1.99
EPG
C
EPG A, EPG B & EPG C - Link Level traffic is flooded only to the endpoints within the
EPG
Cisco Public

Disabling Flooding within the BD
EPG
B
EPG
A
100.1.1.3
TECACI-2009
100.1.1.99
100.1.1.72
100.1.1.7
100.1.1.5
EPG
C
100.1.1.3
When desired each Bridge Domain can be configure to permit or disable

flooding of Link Local, BCAST and L2 MCAST traffic
Cisco Public
An Example of
Interconnecting and Migrating
Many Different Physical Designs

N7k
Cat6500
L3 HSRP
L3 HSRP
Logical Design
FEX
P
P
HSRP
Default GW
VM
VM
N7k
VLAN / Subnet
L3 HSRP
N7k
L3 HSRP
VM
VM
vPC
VM
vPC
N5k
N2k
P
TECACI-2009
Cisco Public
N5k
VM
VM
Connect Fabric to existing Network

Functionally we are expanding the VLANs into ACI.
Existing Design
ACI Fabric
HSRP
Default GW
VLAN 10 / Subnet 10
EPG-10 = VLAN 10
TECACI-2009
VM
VM
VM
Cisco Public
Configure ACI Bridge Domain settings

Tenant Red
Temporary Bridge Domain

specific settings while we are
using the HSRP gateways in
the existing network.
Context Red
Bridge Domain 10
Subnet 10
EPG-10
Select Forwarding to be
Custom which allow
Enable Flooding of L2
unknown unicast
Enble ARP flooding
Disable Unicast routing
TECACI-2009
Cisco Public
Migrate Workloads
APIC point of view, the policy model
EPG 10
P
VM
VM
APIC
VMs will need to be connected to new Port

Group under APIC control (AVS or DVS).
Existing Design
HSRP
Default GW
VLAN 10 / Subnet A
P
TECACI-2009
VM
VM
Cisco Public
VM
VM
Complete the Migration

Change BD settings back to normal for ACI mode
Change BD settings back to default.
No Flooding
Unicast Routing enabled.
TECACI-2009
Cisco Public
Migrating Default Gateway to the ACI Fabric
Change GW MAC address. By default, All

fabric and all BD share same GW MAC
Enable Routing and ARP flooding
TECACI-2009
Cisco Public
Extension and Connecting

Its a Network with any VLAN Anywhere
Anycast Default Gateway
10.10.10.6 10.20.20.31
10.10.10.8
10.20.20.32
10.10.10.9
Any IP - Anywhere
TECACI-2009
Cisco Public
10.20.20.33
Policy can be added gradually starting with what you have

today
External Networks Application
(Outside)
Client
Redirect to Preconfigured FW
Subnet
Subnet
10.30.30.0/24
Subnet
10.20.20.0/24
10.10.10.0/24
Critical Users
(Outside)
Subnet
10.50.50.0/24
Subnet
10.40.40.0/24
Default Users
(Outside)
Permit TCP any

any
Subnet
10.20.20.0/24
Subnet
10.30.30.0/24
Subnet
10.10.10.0/24
Redirect to
dynamically
configured FW
Web
Servers
BRKACI-2001
Redirect to dynamically
configured FW
Middle Ware
Servers
Cisco Public
NFS
Servers
DB Contract
Oracle
NFS Contract
Simple Policy During Migration - Any-to-Any

Configuration
Contracts
Provided
Filter
Contracts
Provided
Contracts
consumed
Filter
EPG VLAN 10
VLAN10
Default
ALL
ALL
Default
EPG VLAN 20
VLAN20
Default
ALL
ALL
EPG VLAN 30
VLAN30
Default
ALL
ALL
ALL
VLAN 10
VLAN 20
BRKACI-2001
Cisco Public
VLAN 30
I want to have a very open configuration with VLAN10

talking to anything (1)
Create Contract
ALL if it doesnt exist
yet
Use filter
common/default
BRKACI-2001
Cisco Public
I want to have a very open configuration with VLAN10

talking to anything (2)
EPG VLAN 10
provides and
consumes ALL
BRKACI-2001
Cisco Public

Dynamic Distributed ACLs
Permit ACL is applied on
all ports between VLAN
10, 20 & 30
10.10.10.6 10.20.20.31
10.10.10.8
10.20.20.32
10.10.10.9
10.20.20.33
All Subnets are allowed to communicate with this policy applied
BRKACI-2001
Cisco Public
Later if I want to put an ACL between VLAN 10 and 20

Contracts
Provided
Filter
EPG VLAN 10
VLAN10
Default
EPG VLAN 20
VLAN20
Default
EPG VLAN 30
VLAN30
Default
Contracts
Provided
ALL
VLAN 10
VLAN 20
BRKACI-2001
Cisco Public
Contracts
consumed
Filter
VLAN20
Port 80
ALL
ALL
Default
ALL
ALL
VLAN 30

Dynamic ACLs
Dynamic ACL is applied
between all endpoints
only allowing port 80
10.10.10.6 10.20.20.31
10.10.10.8
10.20.20.32
10.10.10.9
10.20.20.33
Traffic is controlled between VLAN 10 & 20 to HTTP (port 80)
BRKACI-2001
Cisco Public
Integrating and Extending L4-7 Services
Automate Service Insertion Through APIC

Application profile
EXTERNAL
Policy
WEB
Policy
APP
Policy
DB
APIC Policy Model

Endpoint Group (EPG): Collection of similar End Points identifying a particular
Application Tier. Endpoint could represent VMs, VNICs , IP, DNS name etc
Application Profile: Collection of Endpoint Groups and the policies that define way
Endpoint group communicate with each other
BRKACI-2001
Cisco Public
ACI Service Insertion via Policy

EPG
1
Application
Admin
Packet match on a
redirection rule sends the
packet into a services
graph.
Stage
1
Stage
2
Service
Admin
Service graph simplifies and

scales service operations
ASA
5585
Cisco Public
EPG
2
Chain
FW_ADC 1
Begin
Service Graph can be one

or more service nodes predefined in a series.
BRKACI-2001
Policy-based
Redirection
Netscaler
VPX
End
FW_ADC 1
Automated and scalable L4L7 service insertion
Service Automation Through Device Package

Open Device
Package
Configuration
Model (XML File)
APIC
Policy
Engine
APIC provides extendable policy model through Device Package
Call Back Script
Device Package contains XML fine defining Device Configuration

APIC Policy Manager
Configuration
Model
Provider Administrator can upload a Device Package
Event Engine
APIC Script Interface
Call Back Scripts
Device scripts translates APIC API callouts to device specific callouts

Device Interface: REST/CLI
BRKACI-2001
Cisco Public

Standard Architecture with Services
Backbone
Services Chassis
Services Chassis
vSwitch
BRKACI-2001
vSwitch
Cisco Public
vSwitch

Services Switch
Dynamic ACL is applied
between all endpoints
only allowing port 80
10.10.10.6 10.20.20.31
10.10.10.8
10.20.20.32
10.10.10.9
Start with the picture we now understand
BRKACI-2001
Cisco Public
10.20.20.33

Services Switch
10.10.10.8
10.20.20.32
Lets simplify things
BRKACI-2001
Cisco Public

Services Switch
10.10.10.8
10.20.20.32
Start with just a few switches
BRKACI-2001
Cisco Public

Services Switch
10.10.10.8
10.20.20.32
Lets connect them the way we should, redundantly (the previous slides showed a simple interconnect
just as an example
BRKACI-2001
Cisco Public

Services Switch
10.10.10.8
10.20.20.32
Lets use a small spine switch

We are starting with a very small fabric
BRKACI-2001
Cisco Public

Services Switch
Same picture as seen if the fabric was a services switch
BRKACI-2001
Cisco Public

Services Switch
Attach the L4-7 Services

They can be physical and/or virtual
BRKACI-2001
Cisco Public

Services Switch
vSwitch
vSwitch
vSwitch
Add in the Core/WAN and the Servers and vSwitches

BRKACI-2001
Cisco Public

Services Switch
vSwitch
vSwitch
vSwitch
Activate the services, leverage the services chaining and dynamic provisioning
Leverage the fabric as the layer 3 gateway for all the other VLANs
BRKACI-2001
Cisco Public
Extending ACI - Application Virtual Switch
Hypervisor Integration with ACI

ACI Fabric implements policy on
Virtual Networks by mapping

Endpoints to EPGs
APIC
Endpoints in a Virtualised
environment are represented as the

vNICs
VMM applies network configuration
Application Network Profile

F/W
WEB PORT GROUP
EPG
APP
L/B
APP PORT GROUP
VM
BRKACI-2001
EPG
WEB
VM
by placement of vNICs into Port

Groups or VM Networks
EPG
DB
EPGs are exposed to the VMM as a
DB PORT GROUP
1:1 mapping to Port Groups or VM

Networks
VM
Cisco Public
79
VMWare Integration
Three Different Options
Distributed Virtual Switch
(DVS)
vCenter + vShield
Application Virtual Switch

(AVS)
+
Encapsulations: VLAN
Installation: Native
VM discovery: LLDP
Software/Licenses:
vCenter with
Enterprise+ License
BRKACI-2001
Encapsulations: VLAN,
VXLAN
Encapsulations: VLAN,
VXLAN
Installation: Native
Installation: VIB through

VUM or Console
VM discovery: LLDP
Software/Licenses:
vCenter with
Enterprise+ License,
vShield Manager with
vShield License
Cisco Public
80
VM discovery: OpFlex
Software/Licenses:
vCenter with
Enterprise+ License
ACI Hypervisor Integration VMware DVS/vShield

5
APIC
EPG
WEB
F/W
Create Application Policy
EPG
APP
L/B
EPG DB
APIC Admin
ACI
Fabric
Push Policy
1
Cisco APIC and
VMware vCenter Initial
Handshake
Create VDS
Create Port
Groups
VIRTUAL DISTRIBUTED SWITCH

WEB PORT GROUP
vCenter
Server / vShield
8
VI/Server Admin
Instantiate VMs,
Assign to Port Groups
Attach Hypervisor
to VDS
Web
App
HYPERVISOR
BRKACI-2001
Learn location of ESX

Host through LLDP
Automatically Map
EPG To Port Groups
Cisco Public
81
APP PORT GROUP
DB
Web
DB PORT GROUP
Web
HYPERVISOR
DB
ACI Hypervisor Integration VMware DVS

Name of VMM Domain
Type of vSwitch (DVS or AVS)
Associated Attachable Entity Profile (AEP)
VLAN Pool
vCenter Administrator Credentials
vCenter server information
BRKACI-2001
Cisco Public
82
ACI Hypervisor Integration VMware DVS
BRKACI-2001
Cisco Public
83
Application Virtual Switch with OpFlex in ACI Fabric

AVS: First Virtual Leaf to
implement OpFlex
Hypervisor Manager
Network policy communicated

from APIC to AVS through N9k
using OpFlex
vCenter
OpFlex
Increased control plane scale

through APIC Cluster and Leaf
Node
APIC communicates with vCenter
Server for Port Group creation
OpFlex
OpFlex
VM
VM
OpFlex
VM
VM
AVS
BRKACI-2001
Cisco Public
VM
VM
VM
VM
AVS
Hypervisor Integration with ACI

Endpoint Discovery
Virtual Endpoints are
APIC
discovered for reachability &

policy purposes via 2 methods:
Control Plane Learning:
-
Out-of-Band Handshake: vCenter

APIs
Inband Handshake: OpFlexenabled Host (AVS, Hyper-V,

etc.)
Data Path Learning: Distributed
switch learning
Control
(vCenter API)
Control
(OpFlex)
LLDP used to resolve Virtual
host ID to attached port on leaf

node (non-OpFlex Hosts)
BRKACI-2001
Data Path
Data Path
DVS Host
OpFlex Host
Cisco Public
85
VMM
ACI Hypervisor Integration AVS

5
APIC
EPG
WEB
F/W
EPG
APP
L/B
EPG
DB
APIC Admin
ACI
Fabric
Push Policy
1
Cisco APIC and
VMware vCenter Initial
Handshake
Learn location of ESX

Host through OpFlex
Automatically Map
EPG To Port Groups
OpFlex Agent
Create AVS
VDS
Create Port
Groups
Application Virtual Switch (AVS)

WEB PORT GROUP
vCenter
Server
8
VI/Server Admin
Instantiate VMs,
Assign to Port Groups
Attach Hypervisor
to VDS
Web
App
HYPERVISOR
BRKACI-2001
Cisco Public
86
OpFlex Agent
APP PORT GROUP
DB
Web
DB PORT GROUP
Web
HYPERVISOR
DB
ACI Hypervisor Integration Cisco AVS

Name of VMM Domain
Type of vSwitch (DVS or AVS)
Switching mode (FEX or Normal)
Associated Attachable Entity Profile (AEP)
VXLAN Pool
Multicast Pool
vCenter Administrator Credentials
vCenter server information
BRKACI-2001
Cisco Public
87
Extending ACI to Existing Virtual & Physical Networks

VLAN & VXLAN Extension
Layer 2
AVS
AVS
OpFlex
AVS
OpFlex
AVS
Layer 2
AVS
AVS
AVS supports OpFlex to integrate with APIC
Supports a Full multi-hop Layer 2 Network between Nexus 9k and AVS: Investment Protection
Layer 2 network is required to support OpFlex bootstrapping in this phase
BRKACI-2001
Cisco Public
Extending ACI to Existing Virtual & Physical Networks

VLAN & VXLAN Extension
Layer 2
AVS
AVS
OpFlex
AVS
OpFlex
AVS
Layer 2
AVS
Supports VLAN and VXLAN for transport (Recommend VXLAN to automate new workload)
Existing Network need to have 1 Infrastructure VLAN for VXLAN transport
Multicast: Turn on IGMP Snooping
Recommend 1 L2 Multicast group per EPG
BRKACI-2001
Cisco Public
AVS
ACI - Investment protection for INTEGRATION / MIGRATION

Remote VTEP (Virtual) via AVS
Spine 1
Spine 2
Fabric
(40Gbps)
Leaf 1
Leaf 2
APIC
APIC
VXLAN
VLAN
ESXi-1
N1
Kv
lan
-vS
Co
i-W
eb
ke
-W
eb
Co
ke
-A
pp
ESXi-1
Pe
ps
i-a
p
pe
ps
lea
f1
ESXi-2
ESXi-3
N1
k-A
PP
N1
k-W
EB
EP
G1
ar
ed
_
Sh
pe
ps
i-d
b
ESXi-4
Nexus 3K (L2)
wi
tch
-
VLAN
APIC
Leaf 3
BRKACI-2001
Cisco Public
ACI Hypervisor Integration VMware
BRKACI-2001
Cisco Public
91
ACI Azure Pack Integration

1
APIC
APIC Admin
(Basic Infrastructure)
ACI
Fabric
3
Get VLANs allocated
for each EPG
Pull Policy on leaf

where EP attaches
Push Network
Profiles to APIC
6
5
Create VM Networks
Create Application
Policy
4
1
Indicate EP Attach to attached leaf

when VM starts
Instantiate VMs
4
APIC Plugin
SCVMM Plugin
OpFlex Agent
OpFlex Agent
HYPERVISOR
OpFlex Agent
HYPERVISOR
HYPERVISOR
Azure Pack \ SPF

Azure Pack Tenant
BRKACI-2001
Web
Cisco Public
App
Web
App
DB
Web
Web
DB
ACI OpenStack Integration

Create Application Network
Profile
NEUTRON
NOVA
OpenStack Tenant
(Performs step 1,4)
Web
Instantiate VMs
EPG
WEB
F/W
L/B
App
Web
HYPERVISOR
EPG
APP
L/B
DB
App
Web
HYPERVISOR
EPG
DB
DB
Web
HYPERVISOR
Automatically Push
Network Profiles to
APIC

3
APIC
ACI Admin
(manages physical
network, monitors tenant
state)
5
Push Policy
BRKACI-2001
F/W
L/B
Cisco Public
ACI
Fabric
EPG
WEB
L/B
EPG
APP
EPG
DB
Hypervisors vs. Linux Containers

Containers share the OS kernel of the host and thus are lightweight.
However, each container must have the same OS kernel.
Containers are isolated, but share
OS and, where appropriate, libs /
bins.
App
App
App
App
App
Bins / libs
Bins / libs
Operating
System
Operating
System
Virtual Machine
Virtual Machine
App
App
Bins / libs
Bins / libs
Operating
System
Operating
System
Virtual Machine
Virtual Machine
Hypervisor
App
App
App
Bins / libs
Container
Bins / libs
Operating System
Operating System
Hardware
Hardware
Hardware
Type 2 Hypervisor
Cisco Public
App
Container
Hypervisor
Type 1 Hypervisor
BRKACI-2001
App
Linux Containers (LXC)
Docker and ACI
http://www.cisco.com/c/en/us/solutions/collater
al/data-center-virtualization/application-centricinfrastructure/white-paper-c11-732697.html
BRKACI-2001
Cisco Public
One Network for Everything - Logical

External
Zone
EXTERNAL
DMZ ACI
Policy
Virtual Machines
FW
ADC
WEB
Docker Containers
Trusted
ACI
Zone
APP
Policy
DB
ACI
Tier
Policy
Bare-Metal Server
DB
SECURITY
HYPERVISOR
BRKACI-2001
Cisco Public
96
HYPERVISOR
HYPERVISOR
96
One Network for Everything
3/1
3/9
3/3
1/97
1/1
1/97
1/2
1/3
KVM
1/2
1/3
1/4
Docker
ESX1
BRKACI-2001
1/1
1/97
Cisco Public
1/6
KVM
ESX2
LxC
1/2
1/1
HyperV
BareMetal
ESX3
Docker
Extending ACI into Existing Data Centres

Adding Remote Switch Nodes
Integration of Existing DC Network Assets

Extending the ACI Overlay to Existing DC Assets
vSwitch
vSwitch
vSwitch
Maintain Existing Physical Network infrastructure and operations

Extend the ACI policy based forwarding to Hypervisor environment
attached to the existing physical network
BRKACI-2001
Cisco Public
IP Core with a Hardware Directory Service

Transit Node
Multicast Root
Proxy Lookup
Proxy Lookup
Unknown EID
Cached EID
Entry
Three data plane functions required within the core of an ACI fabric
Transit: IP forwarding of traffic between VTEPs
Multicast Root: Root for one of the 16 multicast forwarding topologies (used for optimisation of multicast load
balancing and forwarding)
Proxy Lookup: Data Plane based directory for forwarding traffic based on mapping database of EID to VTEP bindings
Not all functions are required on all spine switches
BRKACI-2001
Cisco Public
ACI Spine Proxy == LISP Proxy Tunnel Router

+ Map-Server
http://tools.ietf.org/html/draft-moreno-lisp-datacenter-deployment-00
BRKACI-2001
Cisco Public
Extension of the ACI Overlay to remote AVS

ACI Extended Overlay CY15
ACI VXLAN Extended
Overlay
Infrastructure VRF
Extended
VTEP
VTEP
VTEP
VTEP
L2 or L3
Lets Look at
the Details
Direct Attach
Endpoints
vSwitch
Hypervisor Attached
Endpoints (VLAN or
VXLAN)
VTEP
AVS
ACI Policy overlay can be extended over existing IP

networks
BRKACI-2001
Cisco Public
VTEP
AVS
Full ACI VXLAN

Switching Enabled
Hypervisor
Forwarding within the Extended Overlay

Adding Remote Leaf Nodes, Nexus 9000
ACI VXLAN Extended
Overlay
Infrastructure VRF
Extended
VTEP
VTEP
VTEP
VTEP
VTEP
AVS
VM
VM
10.9.3.123
10.2.4.19
VTEP
VTEP
vSwitch
AVS
VM
VTEP
VM
VTEP
10.9.3.38 10.2.4.32
VM
Support for full policy based forwarding, atomic counters,

zero touch install, health scores
DVS
AVS
Nexus 9000 as a remote ACI Leaf
BRKACI-2001
VTEP
Lets Look at
the Details
Cisco Public
VM
10.9.3.37 10.2.4.7
VM
10.9.3.89
VM
10.2.4.74

VTEP
IP
VTEP
VTEP
Group
Policy
VNID
Tenant Packet
VTEP
VTEP
VTEP
AVS
VM
VM
10.9.3.123
10.2.4.19
VTEP
VTEP
vSwitch
AVS
VM
VTEP
VM
VTEP
10.9.3.38 10.2.4.32
VM
Support for full policy based forwarding, atomic counters,

zero touch install, health scores
DVS
AVS
Nexus 9000 as a remote ACI Leaf
BRKACI-2001
VTEP
Cisco Public
VM
10.9.3.37 10.2.4.7
VM
10.9.3.89
VM
10.2.4.74

VTEP
VTEP
VTEP
VTEP
VTEP
AVS
VM
VM
10.9.3.123
10.2.4.19
VTEP
VTEP
vSwitch
AVS
VM
VTEP
VTEP
VM
VTEP
10.9.3.38 10.2.4.3
2
AVS
Nexus 9000 as a remote ACI Leaf will support vSwitch

downstream (ESX/DVS, Hyper-V, OVS)
Leverage Existing Hypervisor implementations
BRKACI-2001
VTEP
Cisco Public
VM
VM
10.9.3.37 10.2.4.7
vSwitch
DVS
VM
10.2.4.74
VM
10.9.3.89

VTEP
VTEP
VTEP
VTEP
VTEP
AVS
VM
VM
10.9.3.123
10.2.4.19
VTEP
VTEP
vSwitch
AVS
VM
VTEP
VTEP
VM
VTEP
10.9.3.38 10.2.4.32
AVS
ACI 9300 performs both L2 and L3 forwarding (Remote Leaf

is a full member of the fabric)
Remote 9300 Leaf performs full local inter EPG policy

forwarding
BRKACI-2001
VTEP
Cisco Public
VM
VM
10.9.3.37 10.2.4.7
vSwitch
DVS
VM
10.2.4.74
VM
10.9.3.89

Unknown MAC/IP
Known MAC/IP
(Cache Entry Exists)
VTEP
VTEP
VTEP
VTEP
VTEP
AVS
VM
VM
10.9.3.123
10.2.4.19
VTEP
VTEP
vSwitch
AVS
VM
VTEP
VTEP
VM
VTEP
10.9.3.38 10.2.4.3
2
AVS
First packet for an unknown MAC or IP host address passes

through the inline directory
Subsequent packets are forwarded directly to the target
VTEP (local switching)
BRKACI-2001
VTEP
Cisco Public
VM
VM
10.9.3.37 10.2.4.7
vSwitch
DVS
VM
10.2.4.74
VM
10.9.3.89

VTEP
VTEP
VTEP
VTEP
VTEP
VTEP
vSwitch
AVS
VM
VTEP
VTEP
VM
VTEP
10.9.3.38 10.2.4.32
AVS
ACI services insertion and full L4-L7 policy are supported via
the AVS
BRKACI-2001
VTEP
Cisco Public
VM
VM
10.9.3.37 10.2.4.7
VTEP
AVS
VM
10.2.4.74
vSwitch
VM
10.9.3.89
Multi-Site Fabrics
VTEP
IP
Group
Policy
VNID
Tenant Packet
Fabric A
Fabric B
Multi-Site
Traffic
mBGP - EVPN
DB
Web/App
Web/App
Host Level Reachability Advertised between Fabrics via BGP
Transit Network is IP Based
Host Routes do not need to be advertised into transit network
Policy Context is carried with packets as they traverse the transit IP Network
Forwarding between multiple Fabrics is allowed (not limited to two sites)
BRKACI-2001
Cisco Public
Applications will spread across existing and new

infrastructure
Web
Outside
(Tenant
VRF)
App
DB
QoS
QoS
QoS
Service
Filter
Filter
Outside
(Tenant VRF)
Web
DB
vPC
App
QoS
QoS
Service
Filter
vPC
vPC
DB Outside EPG
QoS
Filter
BRKACI-2001
Cisco Public
ACI
Its an IP Network
Directory/Proxy
Service Nodes
ACI Enabled L4-7
Virtual and Physical
Services (Support for
Existing and New
Services Instances)
Border Leaves
IP Enabled Data Centre

Network
APIC Policy
Controller
ACI Leaf
Nexus 9000
AVS
vSwitch
ACI Policy and Automation

Extended to Physical and
Existing Virtual Servers via
Cisco Nexus 9000
AVS
ACI Virtual Leaf

(AVS)
Extending ACI Policy and Automation into the Existing Data Centre
BRKACI-2001
ACI Services
extended in to any
existing IP
enabled Data
Centre
Cisco Public
ACI Policy and Automation

Extended to Virtual Servers
via Cisco AVS
Recommended Readings
BRKACI-2001
Cisco Public
113
Complete Your Online Session Evaluation

Give us your feedback and receive a
Cisco Live 2015 T-Shirt!
Complete your Overall Event Survey and 5 Session
Evaluations.
Directly from your mobile device on the Cisco Live
Mobile App
By visiting the Cisco Live Mobile Site
http://showcase.genie-connect.com/clmelbourne2015
Visit any Cisco Live Internet Station located
throughout the venue
T-Shirts can be collected in the World of Solutions
on Friday 20 March 12:00pm - 2:00pm
BRKACI-2001
Cisco Public
Learn online with Cisco Live!

Visit us online after the conference for full
access to session videos and
presentations. www.CiscoLiveAPAC.com

BRKACI-2001 - Integration and Interoperation of Existing Nexus Networks Into An ACI Architecture (2015 Melbourne)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKACI-2001 - Integration and Interoperation of Existing Nexus Networks Into An ACI Architecture (2015 Melbourne)

Uploaded by

Copyright:

Available Formats

Integration and Interoperation of

Existing Nexus Networks into an ACI

Mike Herbert Principal Engineer INSBU

Introducing: Application Centric Infrastructure (ACI)

Application Oriented Policy = Operational Simplicity

2014 Cisco and/or its affiliates. All rights reserved.

Why Networks are Complex

Redirect and Load Balance

Control & Audit Connectivity

2014 Cisco and/or its affiliates. All rights reserved.

Application Specific Connectivity

Why Network Provisioning is Slow

Developer and infrastructure teams must translate between disparate languages.

2014 Cisco and/or its affiliates. All rights reserved.

Centralised Policy and Distributed Enforcement

2014 Cisco and/or its affiliates. All rights reserved.

Two Big Questions

Do I need to replace all of my existing

ABSOLUTELY NOT !!!

2014 Cisco and/or its affiliates. All rights reserved.

Things we would like to understand how to do

Let me just run

Extend ACI to local

2014 Cisco and/or its affiliates. All rights reserved.

Extend ACI to to existing Nexus

ACI Policy Based Forwarding via an

ACI Fabric An IP network with an Integrated Overlay

Ciscos ACI solution leverages an integrated VXLAN based overlay

IP Network for Transport

VXLAN based tunnel end points (VTEP)

VTEP discovery via infrastructure routing

ACI Fabric Integrated Overlay

Interworking with ACI

VXLAN enabled Hypervisor (FCS)

VXLAN Hardware VTEP (Nexus 9000 standalone, Nexus 3100/7000-F3, ASR9K, )

2014 Cisco and/or its affiliates. All rights reserved.

ACI Fabric Integrated Overlay

All Tenant traffic within the Fabric is

Policy attributes are carried by every packet

Policy Group (source group)

Forwarding Group (Tenant, VRF,

Load Balancing Policy

ACI VXLAN (VXLAN) header identifies the

At the ingress port the Fabric translates

2014 Cisco and/or its affiliates. All rights reserved.

External Identifiers are localised to

ACI leverages VXLAN

2014 Cisco and/or its affiliates. All rights reserved.

Overlay Elements Host Forwarding

Forwarded using DIPi

Forwarded using DMAC

Route if MAC is router-mac, otherwise bridge (standard L2/L3 behaviour)

2014 Cisco and/or its affiliates. All rights reserved.

Location Independent Forwarding

Distributed Default Gateway

Directed ARP Forwarding

2014 Cisco and/or its affiliates. All rights reserved.

Where is my default Gateway?

External Default Gateway

Default Gateway can reside internal or external to the Fabric

2014 Cisco and/or its affiliates. All rights reserved.

Overlay Elements - Mapping/Directory Proxy

Global Station Table