IPexpert CCIE Data Center Volume 1 Detailed Solution Chapters 1 8 PDF

IPexperts Lab Preparation Detailed Solution Guide
for the Cisco CCIE Data Center v1.0 Lab Exam

Volume 1
Authored by: Rick Mur - CCIE3 #21946 (R&S / SP / Storage), JNCIE-SP #851
CCIE Data Center Lab Preparation Detailed Solution Guide
IPexperts
Lab Preparation Detailed Solution Guide for
Ciscos CCIE Data Center Lab
Before We Begin
This product is part of the IPexpert suite of materials that provide CCIE candidates and network
engineers with a comprehensive training program. For information about the full solution, contact an
IPexpert Training Advisor today.
Telephone: +1.810.326.1444
Email: sales@ipexpert.com
Congratulations! You now possess one of the ULTIMATE CCIETM Lab preparation and network
operation resources available today! This resource was produced by senior engineers, technical
instructors, and author boasting decades of internetworling experience. Although there is no way to
100% guarantee success rate on the CCIE Data Center Lab exam, we feel VERY confident that your
chances of passing the Lab will improve dramatically after completing this industry-recognized
Workbook!
Technical Support from IPexpert, and your CCIE community!
Copyright by IPexpert. All rights reserved.
IPexpert is proud to lead the industry with multiple support options at your disposal free of charge. Our
online communities have attracted a membership of over 20,000 of your peers from around the world!
At blog.ipexpert.com, you can keep up to date with everything IPexpert does and read the latest in
technical articles from world-renowned IPexpert instructors. At OnlineStudyList.com, you may subscribe
to multiple SPAM-free, moderated CCIE-focused email lists.
Feedback
Do you have a suggestion or other feedback regarding this book or other IPexpert products? At IPexpert,
we look to you our valued clients for the real world, frontline evaluation that we believe is necessary
so that we may always improve. Please send an email with your thoughts to feedback@ipexpert.com or
call 1.866.225.8064 (international callers dial +1.810.326.1444).
In addition, for those using this book as CCIETM preparation, when you pass the CCIETM Lab exam, we
want to hear about it! Email your CCIETM number to success@ipexpert.com and let us know how
IPexpert helped you succeed. We would like to send you a gift of thanks and congratulations.
Additional CCIETM Preparation Material

IPexpert, Inc. is committed to developing the most effective Cisco CCIETM R&S, Security, Voice, Wireless
and Data Center Lab certification preparation tools available. Our team of certified networking
professionals develops the most up-to-date and comprehensive materials for networking certification,
including self-paced workbooks, online Cisco hardware rental, classroom training, online (distance
learning) instructor-led training, audio products, and video training materials. Unlike other certificationtraining providers, we employ the most experienced and accomplished teams of experts to create,
maintain, and constantly update our products. At IPexpert, we are focus on making your CCIETM Lab
preparation more effective.
Issues with this Book

This book is carefully edited to ensure the accuracy of all content. Should you find any error whatsoever,
please email a page reference and detailed comment to wberrors@ipexpert.com. Your email will be
responded to promptly.
IPEXPERT END-USER LICENSE AGREEMENT

END USER LICENSE FOR ONE (1) PERSON ONLY
IF YOU DO NOT AGREE WITH THESE TERMS AND CONDITIONS,
DO NOT OPEN OR USE THE TRAINING MATERIALS.
This is a legally binding agreement between you and IPEXPERT, the Licensor, from whom you have
licensed the IPEXPERT training materials (the Training Materials). By using the Training Materials, you
agree to be bound by the terms of this License, except to the extent these terms have been modified by
a written agreement (the Governing Agreement) signed by you (or the party that has licensed the
Training Materials for your use) and an executive officer of Licensor. If you do not agree to the License
terms, the Licensor is unwilling to license the Training Materials to you. In this event, you may not use
the Training Materials, and you should promptly contact the Licensor for return instructions.
The Training Materials shall be used by only ONE (1) INDIVIDUAL who shall be the sole individual
authorized to use the Training Materials throughout the term of this License.
Copyright and Proprietary Rights

The Training Materials are the property of IPEXPERT, Inc. ("IPEXPERT") and are protected by United
States and International copyright laws. All copyright, trademark, and other proprietary rights in the
Training Materials and in the Training Materials, text, graphics, design elements, audio, and all other
materials originated by IPEXPERT at its site, in its workbooks, scenarios and courses (the "IPEXPERT
Information") are reserved to IPEXPERT.
The Training Materials cannot be used by or transferred to any other person. You may not rent, lease,
loan, barter, sell or time-share the Training Materials or accompanying documentation. You may not
reverse engineer, decompile, or disassemble the Training Materials. You may not modify, or create
derivative works based upon the Training Materials in whole or in part. You may not reproduce, store,
upload, post, transmit, download or distribute in any form or by any means, electronic, mechanical,
recording or otherwise any part of the Training Materials and IPEXPERT Information other than printing
out or downloading portions of the text and images for your own personal, non-commercial use without
the prior written permission of IPEXPERT.
You shall observe copyright and other restrictions imposed by IPEXPERT. You may not use the Training
Materials or IPEXPERT Information in any manner that infringes the rights of any person or entity.
Exclusions of Warranties
THE TRAINING MATERIALS AND DOCUMENTATION ARE PROVIDED AS IS. LICENSOR HEREBY DISCLAIMS
ALL OTHER WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING WITHOUT LIMITATION, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. SOME STATES
DO NOT ALLOW THE LIMITATION OF INCIDENTAL DAMAGES OR LIMITATIONS ON HOW LONG AN
IMPLIED WARRANTY LASTS, SO THE ABOVE LIMITATIONS OR EXCLUSIONS MAY NOT APPLY TO YOU. This
agreement gives you specific legal rights, and you may have other rights that vary from state to state.
Choice of Law and Jurisdiction

This Agreement shall be governed by and construed in accordance with the laws of the State of
Michigan, without reference to any conflict of law principles. You agree that any litigation or other
proceeding between you and Licensor in connection with the Training Materials shall be brought in the
Michigan state or courts located in Port Huron, Michigan, and you consent to the jurisdiction of such
courts to decide the matter. The parties agree that the United Nations Convention on Contracts for the
International Sale of Goods shall not apply to this License. If any provision of this Agreement is held
invalid, the remainder of this License shall continue in full force and effect.
Limitation of Claims and Liability

ANY ACTION ON ANY CLAIM AGAINST IPEXPERT MUST BE BROUGHT BY THE USER WITHIN ONE (1) YEAR
FOLLOWING THE DATE THE CLAIM FIRST ACCRUED, OR SHALL BE DEEMED WAIVED. IN NO EVENT WILL
THE LICENSORS LIABILITY UNDER, ARISING OUT OF, OR RELATING TO THIS AGREEMENT EXCEED THE
AMOUNT PAID TO LICENSOR FOR THE TRAINING MATERIALS. LICENSOR SHALL NOT BE LIABLE FOR ANY
SPECIAL, INCIDENTAL, INDIRECT, OR CONSEQUENTIAL DAMAGES, HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, REGARDLESS OF WHETHER LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES. WITHOUT LIMITING THE FOREGOING, LICENSOR WILL NOT BE LIABLE FOR LOST
PROFITS, LOSS OF DATA, OR COSTS OF COVER.
Entire Agreement
This is the entire agreement between the parties and may not be modified except in writing signed by
both parties.
U.S. Government - Restricted Rights

The Training Materials and accompanying documentation are commercial computer Training
Materials and commercial computer Training Materials documentation, respectively, pursuant to
DFAR Section 227.7202 and FAR Section 12.212, as applicable. Any use, modification, reproduction
release, performance, display, or disclosure of the Training Materials and accompanying documentation
by the U.S. Government shall be governed solely by the terms of this Agreement and shall be prohibited
except to the extent expressly permitted by the terms of this Agreement.
IF YOU DO NOT AGREE WITH THE ABOVE TERMS AND CONDITIONS, DO NOT OPEN OR USE THE
TRAINING MATERIALS AND CONTACT LICENSOR FOR INSTRUCTIONS ON RETURN OF THE TRAINING
MATERIAL
Contents
IPexperts ...................................................................................................................................................... 1
Lab Preparation Detailed Solution Guide for Ciscos CCIE Data Center Lab............................................. 1
Before We Begin ....................................................................................................................................... 1
Feedback................................................................................................................................................... 2
Additional CCIETM Preparation Material ................................................................................................... 2
Issues with this Book ................................................................................................................................ 2
IPEXPERT END-USER LICENSE AGREEMENT.............................................................................................. 3
Copyright and Proprietary Rights ............................................................................................................. 3
Exclusions of Warranties .......................................................................................................................... 4
Choice of Law and Jurisdiction ................................................................................................................. 4
Limitation of Claims and Liability.............................................................................................................. 4
Entire Agreement ..................................................................................................................................... 5
U.S. Government - Restricted Rights ........................................................................................................ 5
Default Lab Topology ................................................................................................................................ 9
Default passwords and IP addresses ........................................................................................................ 9
Chapter 1: Introduction to CCIE Data Center.............................................................................................. 10
Who Should Read this Book? ................................................................................................................. 11
How to Use this Book ............................................................................................................................. 11
An Introduction to CCIE Data Center...................................................................................................... 11
Availability .............................................................................................................................................. 12
Written exam .......................................................................................................................................... 12
The current published reading list:......................................................................................................... 12
Lab exam................................................................................................................................................. 13
Software Versions................................................................................................................................... 13
CCIE Storage?.......................................................................................................................................... 13
What about P and A tracks? ................................................................................................................... 13
Troubleshooting ..................................................................................................................................... 14
An Introduction to the Proctor Labs CCIE Data Center hardware rack .................................................. 14
Software Versions................................................................................................................................... 17
Chapter 2: Data Center Networking Layer 2 Infrastructure ....................................................................... 18
(NX-OS) ........................................................................................................................................................ 18
Generic Lab topology.............................................................................................................................. 19
Introduction ............................................................................................................................................ 20
General Rules.......................................................................................................................................... 20
Solutions ................................................................................................................................................. 21
Task 1: General set-up ........................................................................................................................ 21
Task 2: Implement VLANs ................................................................................................................... 26
Task 3: Implement Private-VLANs....................................................................................................... 33
Task 4: Implement Rapid Spanning-Tree protocol ............................................................................. 41
Task 5: Implement Multiple Spanning-Tree protocol ......................................................................... 48
Task 6: Spanning-Tree and UDLD features ......................................................................................... 56
Task 7: Fabric Extenders ..................................................................................................................... 58
Task 8: Misc features .......................................................................................................................... 61
Chapter 3: Data Center Networking Layer 3 Infrastructure (NX-OS) .......................................................... 64
General Rules.......................................................................................................................................... 65
Solutions ................................................................................................................................................. 66
Task 1: Layer 3 topology set-up .......................................................................................................... 66

Task 2: Static routing........................................................................................................................... 75
Task 3: EIGRP....................................................................................................................................... 81
Task 4: OSPF ........................................................................................................................................ 94
Task 5: Redistribution, BFD and ECMP ............................................................................................. 116
Task 6: Layer 3 switching features .................................................................................................... 124
Task 7: FabricPath and OTV .............................................................................................................. 127
Chapter 4: Data Center Networking High Availability (NX-OS) ................................................................. 139
General Rules........................................................................................................................................ 140
Solutions ............................................................................................................................................... 141
Task 1: Topology set-up ........................................................................................................................ 141
Task 2: Port-Channels ....................................................................................................................... 143
Task 3: Virtual Port-Channels (vPCs) ................................................................................................. 166
Task 4: Graceful Restart / Non-Stop Forwarding .............................................................................. 184
Task 5: HSRP ...................................................................................................................................... 188
Task 6: VRRP...................................................................................................................................... 198
Task 7: GLBP ...................................................................................................................................... 202
Task 8: Virtual Port-Channels (vPCs) and FabricPath ........................................................................ 206
Chapter 5: Data Center Storage Networking ............................................................................................ 210
General Rules........................................................................................................................................ 211
Solutions ............................................................................................................................................... 212
Task 1: Initial set-up .......................................................................................................................... 212
Task 2: VSANs .................................................................................................................................... 227
Task 3: Zoning ................................................................................................................................... 244
Task 4: FC Domain ............................................................................................................................. 254
Task 5: Fibre Channel Security Features ........................................................................................... 260
Task 6: Advanced Features ............................................................................................................... 271
Chapter 6: Data Center Storage Networking Extension ........................................................................... 276
General Rules........................................................................................................................................ 277
Solutions ............................................................................................................................................... 278
Task 1: Initial set-up .......................................................................................................................... 278
Task 2: FCIP ....................................................................................................................................... 282
Task 3: FCIP Security ......................................................................................................................... 297
Task 4: SAN Extension Tuner ............................................................................................................ 301
Task 5: iSCSI....................................................................................................................................... 304
Task 6: iSLB........................................................................................................................................ 323
Chapter 7: Data Center Unified Fabric ...................................................................................................... 340
General Rules........................................................................................................................................ 341
Solutions ............................................................................................................................................... 342
Task 1: Native Fibre Channel on Nexus............................................................................................. 342
Task 2: Fibre Channel over Ethernet (FCoE) ..................................................................................... 353
Task 3: Multi hop FCoE ..................................................................................................................... 358
Task 4: FCoE Quality of Service (QoS) ............................................................................................... 371
Task 5: N-Port Virtualization (NPV) and N-Port ID Virtualization (NPIV) .......................................... 381
Task 6: FCoE NPV .............................................................................................................................. 391
Chapter 8: Security Features..................................................................................................................... 393
General Rules........................................................................................................................................ 394
Solutions ............................................................................................................................................... 395
Task 1: Port Security ......................................................................................................................... 395

Task 2: DHCP Snooping, DAI, IP Source Guard.................................................................................. 412
Task 3: Access Control Lists............................................................................................................... 419
Task 4: AAA services.......................................................................................................................... 435
Task 5: 802.1X ................................................................................................................................... 448
Task 6: Cisco TrustSec ....................................................................................................................... 451
Chapter 9: Management Features ............................................................................................................ 455
General Rules........................................................................................................................................ 456
Solutions ............................................................................................................................................... 457
Task 1: Role Based Access Control (RBAC) ........................................................................................ 457
Task 2: Traffic monitoring ................................................................................................................. 470
Task 3: NetFlow................................................................................................................................. 474
Task 4: Management protocols ........................................................................................................ 479
Task 5: Device management ............................................................................................................. 493
Task 6: Smart Call Home and GOLD .................................................................................................. 503
Default Lab Topology
Default passwords and IP addresses
Default management username / password: admin / IPexpert123

Other passwords: ipexpert
Management IP addressing: 172.16.100.0/24

Management Default Gateway: 172.16.100.254
Chapter 1:
Introduction to CCIE
Data Center
Chapter 1: Introduction to CCIE Data Center introduces the team of authors, consultants, and editors
that completed this book and describes the books purpose. This chapter also provides suggestions for
the usage of this written work.
10
Who Should Read this Book?

This workbooks primary audience is for those CCIE candidates that are searching for the most
comprehensive and error-free materials available covering the CCIE Data Center practical lab exam.
These students should possess a home rack of equipment for CCIE-level command-line practice, they
should possess an equipment emulator (for certain parts of the topology), or they should rent
equipment from a company like www.proctorlabs.com. The authors and technical editors exhaustively
tested all of the demonstrations found throughout the technology tasks, troubleshooting- and full-scale
lab exercises against all practice rack options described earlier. Where issues arise with popular
equipment emulators, the text makes note. This book is the most remarkably thorough and technically
accurate book written on the CCIE Data Center lab exam to date.
How to Use this Book

This book breaks all specific CCIE Data Center technologies down on a chapter-by-chapter basis for a
complete and thorough review of this broad set of topics. Each chapter is broken down is various tasks
regarding the subject. Following this, the Detailed Solutions Guide provided with this workbook provides
an intense examination of the operation of the tasks, including key aspects of troubleshooting for the
specific technology. After this, the book presents some of the most common issues that can result with a
particular technology-set, and most importantly, details the simple troubleshooting tools and steps that
succeed for remediation.
The final chapters conclude the book with sample lab scenarios that provide a full scale lab exam as you
will see it when you take the actual test. The Detailed Solutions Guide then provides a well-designed
approach for troubleshooting each major task and offers detailed explanations. The text provides
reference guides for the most popular and powerful show and debug commands for a specific
technology.
Each chapter uses specific initial configurations on the specific chapter. Readers may download initial
configurations, or install them in a simple Graphical User Interface (GUI) on www.proctorlabs.com.
Students are encouraged to follow along on a rack of equipment for every section of every chapter. This
really enhances and strengthens the learning process.
An Introduction to CCIE Data Center

Since the release of the Nexus platform there has been talk about when these platforms were to be
introduced in a CCIE track. With the introduction of UCS in 2009 this became an even higher request
especially since UCS really took off in sales.
11
The scope of the exam is pretty much based on the usual suspects, so in summary you should be aware
of the:
UCS B-series blade systems

UCS C-series rackmount systems connected to UCS Manager via FEX
Virtual Interface Cards (virtualized NICs and HBAs) in all servers
Nexus 7000 with all features like VDC, OTV, FabricPath, etc.
Nexus 5500 with all features like FCoE, FEX
Nexus 2000 connected to either the 5k or the 7k
Nexus 1000V distributed virtual switch in ESX
o There is no mention of any VMware product in the blueprint, so expect ESX and vCenter
to be pre-installed on the UCS blades and FC boot to pre-configured disks
MDS 9222i for connecting FC storage to UCS
ACE appliance
DCNM management software
Availability
The live exam is available from September 1st.

Currently there are no dates when the lab is available.
Written exam
The written exam has an extensive blueprint published to Cisco Learning Network (CLN) including a
reading list.
The current published reading list:

Data Center Fundamentals (ISBN-10: 1-58705-023-4)
NX-OS and Cisco Nexus Switching (ISBN-10: 1-58705-892-8)
Cisco Unified Computing System (UCS) (ISBN-10: 1-58714-193-0)
I/O Consolidation in the Data Center (ISBN-10: 1-58705-888-X)
12
Storage Networking Fundamentals (ISBN-10: 1-58705-162-1)
Please find the extensive blueprint published by Cisco on the bottom of this blog post.
Lab exam
There is not much information available regarding the lab exam. Availability is not mentioned. There is
however information regarding the hardware list and this is an immense list of expensive hardware you
require:
Software Versions
NXOS v6.0(2) on Nexus 7000 Switches

NXOS v4.2(1) on Nexus 1000V
NXOS v5.2(2) on MDS 9222i Switches
UCS Software release 2.0(1x) for UCS-6248 Fabric Interconnect and all UCS systems
Software Release A5(1.0) on ACE4710
Cisco Data Center Manager software v5.2(2)
CCIE Storage?
There are currently no plans for replacing CCIE Storage for CCIE Datacenter. Because of this, there will
not be a large focus on MDS/FC configuration as there is another track for that.
What about P and A tracks?

A CCNA Data Center and CCNP Data Center will be released soon!
13
Troubleshooting
Troubleshooting will be a big part of the exam, which is also pretty clear in the blueprint. There is no
confirmation yet how this will be introduced, either using tickets in the CCIE R&S or just by preconfiguration on the lab. I can imagine that they pre-configured a broken Nexus 1000V on an ESX
installation on one of the JBODs. More information on how this troubleshooting is done will be available
during other Q&A sessions. The implication is that it might be trouble tickets like the CCIE R&S.
An Introduction to the Proctor Labs CCIE Data Center hardware rack

The IPexpert CCIE Data Center rack will support 100% of the features that are tested on the lab! We
have based the topology to be close as possible on the CCIE Data Center rack layout, but have ensured
that all features and functionality is there.
Our CCIE Data Center rack layout is based on the very limited information that has been made available
by Cisco. IPexpert has been in close contact with the people involved in creating this lab exam, and
therefore the layout of the rack is based on some early examples and the published components and
software version blueprint.
As you will see the topology is very much based on a common datacenter design and has more 'static'
layout than other CCIE tracks.
The blueprint specified the following components to be in the lab:
First is the NX-OS Networking equipment.
Nexus7009 (with licensing)

o (1) Sup
o (1) 32 Port 10Gb (F1 Module)
o (1) 32 Port 10Gb (M1 Module)
Nexus5548
Nexus2232
The Nexus 7000 will be configured with VDC's to simulate various different topologies and create
multiple 'core switch' layers within the network.
14
Nexus 5548 will be used as a 'distribution' layer within the datacenter network. The Nexus 2k's can be
configured as FEX for the Nexus 7000; Nexus 5000 and the Fabric Interconnects of the UCS system to
connect the UCS C-series rack mount servers. The VDC's are a major component in the network as the
number of devices is limited and the connectivity is very much based on a best practice design.
The below drawing illustrates an example topology from our new CCIE Data Center lab preparation
workbook which is currently under development.
All these interconnections and switches are based within a single physical chassis with complete
separation of the control and data plane protocols!
Second is the storage networking (SAN) equipment:
Dual attached JBODs = Fibre Channel disks

MDS 9222i (dual fabric)
The MDS switches used in the lab are capable of a ton of features. The blueprint however only describes
certain fibre-channel features which are considered 'basic' features like zoning, VSANs, oversubscription
and ISLs. The other major topic on the blueprint is Fibre Channel Expansion over FCIP and iSCSI. These
features are the IP features supported by the MDS platform. The 1G Ethernet connections are connected
to the Nexus switches for testing the expansion features. Through that connection it's possible to
connect the MDS switches across another connection than Fibre Channel. As the CCIE Storage track is
15
not being replaced by the CCIE Data Center the focus on Storage Networking (SAN) features is not that
big. The major topics are more in the features that aren't tested in any other CCIE track.
The JBODs mentioned in this list represent just plain simple hard-disks that are connected via Fibre
Channel. They are used later as shared storage for the UCS system.
The third major component within the hardware blueprint is the Unified Computing System (UCS).
UCS-6248 Fabric Interconnects

UCS-5108 Blade Chassis
o B200 M2 Blade Servers
o Palo/VIC mezzanine card
o Menlo/Emulex mezzanine card
UCS C200 Series Server = Connected to Fabric Interconnects
o VIC card for C-series
This is based on the C-series rackmount servers, connected to the Fabric Interconnects so the C-series
can also be managed from the central UCS manager the same as the Blade chassis is managed.
The blades are equipped with different NICs. This also means a little different configuration. The VIC
cards are the most interesting ones as they can virtualize NICs to present to the OS.
Ones inside the blades there is a pre-installed VMware ESX(i) environment with a Nexus 1000v
distributed virtual switch. As this is a Cisco lab exam, you are not required to know anything about
VMware. Of course you will need to be able to install this environment in your possible own lab, but
when you step into the lab you will face a pre-installed VMware and 1000V. After that, the switch is not
configured and you are required to configure it.
The final topic on the blueprint is called ANS (Application Networking Services). This means an ACE
appliance is in your lab that you will need to configure. There is not much very interesting going on there
and you will not see a lot of points on that appliance. You will need to know the topics as described on
the lab blueprint and our workbook will focus a whole section on these specific topics.
The last components are used for management. You will not be configuring these devices, but just using
them from your student workstation to access the network.
16
Cisco Catalyst Switch 3750 = management ethernet connections

Cisco 2511 Terminal Server = console lines
What is not mentioned on the hardware blueprint list is that you will also need to be able to configure
(or set-up) the DCNM software as is being given by Cisco when you purchase enough Nexus equipment.
Again this is not extremely difficult, but you need to be aware of the basic configuration items related to
this software.
Software Versions

NXOS v4.2(1) on Nexus 1000v
NXOS v5.2(2) on MDS 9222i Switches
UCS Software release 2.0(1x) for UCS-6248 Fabric Interconnect and UCS system
Software Release A5(1.0) for ACE 4710
Cisco Data Center Manager software v5.2(2)
Above you'll find a reference overview of the used software versions. The exact versions are still
unknown where we might be using newer software versions as our IPexpert lab will be using quite new
hardware for virtualization purposes. Within the Nexus 7000 we will be using the new Supervisor 2E,
meaning that we are able to build 8 VDC's and 1 management VDC meaning we have enough flexibility
for some challenging topologies!
The next chapter of this workbook, Chapter 2: Data Center Networking Layer 2 Infrastructure (NX-OS)
begins with the initial topic on the CCIE Data Center Blueprint regarding layer 2 switching, VLANs,
Private-VLANs, Spanning-Tree and other layer 2 features on the NX-OS platform.
17
Chapter 2: Data
Center Networking
Layer 2
Infrastructure
(NX-OS)
Chapter 2: Data Center Networking Layer 2 Infrastructure (NX-OS) is intended to let you be familiar
with the NX-OS CLI on the Nexus switches and afterwards configure Layer 2 Ethernet features on the
physical Nexus switches within the topology as shown at the beginning of this workbook. We highly
recommend to create your own diagram at the beginning of each lab so you are able to draw on your
own diagram, making it much easier when you step into the real lab. Our devices start with a blank
configuration, which will not be the case when you are in the real lab. Then devices are staged with
configuration containing usernames/passwords, management IP addressing, core IP addressing and
(possible) errors.
18
Generic Lab topology
19
Introduction
This first lab is intended to let you be familiar with the NX-OS CLI on the Nexus switches and afterwards
configure Layer 2 Ethernet features on the physical Nexus switches within the topology as shown at the
beginning of this workbook.
We highly recommend to create your own diagram at the beginning of each lab so you are able to draw
on your own diagram, making it much easier when you step into the real lab.
Our devices start with a blank configuration, which will not be the case when you are in the real lab.
Then devices are staged with configuration containing usernames/passwords, management IP
addressing, core IP addressing and (possible) errors.
General Rules
Try to diagram out the task. Draw your own connections the way you like it
Create a checklist to aid as you work thru the lab
Take a very close read of the tasks to ensure you dont miss any points during grading!
Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this
particular chapter
Estimated Time to Complete:
3 hours
20
Solutions
Task 1: General set-up

The write command is no longer available within NX-OS, though you need to use write erase to
erase the configuration from all boxes. When rebooting from empty configuration you are asked to
configure base defaults to manage the system.
Nexus1# write erase
Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y
Nexus1# reload
This command will reboot the system. (y/n)? [n] y
2012 Aug 26 19:04:24 Nexus1 %$ VDC-1 %$ %PLATFORM-2-PFM_SYSTEM_RESET:
Manual system restart from Command Line Interface
writing reset reason 9,
NX7 SUP Ver 3.22.0
Serial Port Parameters from CMOS
PMCON_1: 0x200
PMCON_2: 0x0
PMCON_3: 0x3a
PM1_STS: 0x101
Performing Memory Detection and Testing
Total mem found : 4096 MB
Performing memory test... Passed.
NumCpus = 2.
Status 61: PCI DEVICES Enumeration Started
Status 62: PCI DEVICES Enumeration Ended
Status 9F: Dispatching Drivers
Status 9E: IOFPGA Found
Status 9A: Booting From Primary ROM
Status 98: Found Cisco IDE
Status 98: Found Cisco IDE
Status 90: Loading Boot Loader
Reset Reason Registers: 0x0 0x10
Filesystem type is ext2fs, partition type 0x83
GNU GRUB
version 0.97
Autobooting bootflash:/n7000-s1-kickstart.5.1.5.bin bootflash:/n7000-s1dk9.5.1

.5.bin...
Filesystem type is ext2fs, partition type 0x83
Booting kickstart image: bootflash:/n7000-s1-kickstart.5.1.5.bin....
..........................................................................
.....
....................Image verification OK
INIT: version 2
Checking all filesystems..r.r.r..r done.
Loading system software
21
/bootflash//n7000-s1-dk9.5.1.5.bin read done

Uncompressing system image: bootflash:/n7000-s1-dk9.5.1.5.bin Sun Aug 26
19:08:10 UTC 2012
blogger: nothing to do.
..done Sun Aug 26 19:08:13 UTC 2012
Load plugins that defined in image conf: /isan/plugin_img/img.conf
Loading plugin 0: core_plugin...
num srgs 1
0: swid-core-supdc3, swid-core-supdc3
num srgs 1
0: swid-supdc3-ks, swid-supdc3-ks
INIT: Entering runlevel: 3
2012 Aug 26 19:08:38 %$ VDC-1 %$ %DAEMON-2-SYSTEM_MSG: <<%PLATFORM-2PS_PWR_INPUT_MISSING>> Power supply 1 present but all AC/DC inputs are
not connected, power redundancy might be affected - platform[2674]
2012 Aug 26 19:08:38 %$ VDC-1 %$ %DAEMON-2-SYSTEM_MSG: <<%PLATFORM-2PS_ABSENT>> Power supply 2 is absent/shutdown, ps-redundancy might be
affected - platform[2674]
2012 Aug 26 19:08:52 %$ VDC-1 %$ %CMPPROXY-2-LOG_CMP_UP: Connectivity
Management processor(on module 9) is now UP
2012 Aug 26 19:08:58 %$ VDC-1 %$ %IDEHSD-2-MOUNT: logflash: online
2012 Aug 26 19:09:09 %$ VDC-1 %$ %COPP-2-COPP_NO_PINST: Control-plane is
unprotected.
2012 Aug 26 19:09:12 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come
online
---- System Admin Account Setup ----
Do you want to enforce secure password standard (yes/no) [y]: 2012 Aug 26
19:09:14 switch %$ VDC-1 %$ %PLATFORM-2-PS_OK: Power supply 1 ok (Serial
number DTM1437011H)
2012 Aug 26 19:09:14 switch %$ VDC-1 %$ %PLATFORM-2-PS_FANOK: Fan in Power
supply 1 ok
2012 Aug 26 19:09:15 switch %$ VDC-1 %$ %PLATFORM-2-XBAR_DETECT: Xbar 1
detected (Serial number JAF1524AKTF)
2012 Aug 26 19:09:23 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 3
detected (Serial number JAF1448BPGN) Module-Type 10 Gbps Ethernet Module
Model N7K-M132XP-12
2012 Aug 26 19:09:23 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 3
powered up (Serial number JAF1448BPGN)
2012 Aug 26 19:09:25 switch %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 5
detected (Serial number JAF1448BBDK) Module-Type 10/100/1000 Mbps
Ethernet Module Model N7K-M148GT-11
2012 Aug 26 19:09:25 switch %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 5
powered up (Serial number JAF1448BBDK)
22
2012 Aug 26 19:09:38 switch %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan

module 1 (Fan1(sys_fan1) fan) ok
2012 Aug 26 19:09:38 switch %$ VDC-1 %$ %PLATFORM-2-FANMOD_FAN_OK: Fan
module 2 (Fan2(sys_fan2) fan) ok
2012 Aug 26 19:11:51 switch %$ VDC-1 %$ %XBAR-2XBAR_INSUFFICIENT_XBAR_BANDWIDTH: Module in slot 3 has insufficient xbarbandwidth.
Enter the password for "admin": IPexpert123

Confirm the password for "admin": IPexpert123
---- Basic System Configuration Dialog VDC: 1 ---This setup utility will guide you through the basic configuration of
the system. Setup configures only enough connectivity for management
of the system.
Please register Cisco Nexus7000 Family devices promptly with your
supplier. Failure to register may affect response times for initial
service calls. Nexus7000 devices must be registered to receive
entitled support services.
Press Enter at anytime to skip a dialog. Use ctrl-c at anytime
to skip the remaining dialogs.
Would you like to enter the basic configuration dialog (yes/no): no
User Access Verification

switch login: admin
Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2011, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch#
Hostnames are configured using the switchname or hostname command. Initially the hostname
command was not available when NX-OS was first forked from SAN-OS. Since this caused a lot of
confusion the hostname command has been ported back into NX-OS. When configured, the one that is
last used will be shown in the configuration. Pay attention to this as this might be a requirement during
the lab.
switch# conf t
Enter configuration commands, one per line.
End with CNTL/Z.

23
switch(config)# hostname SW1-1

SW1-1(config)# sh run | in SW1
hostname SW1-1
vdc SW1-1 id 1
SW1-1(config)# switchname SW1-1
SW1-1(config)# sh run | in SW1
switchname SW1-1
vdc SW1-1 id 1
SW1-1(config)# ip domain-name ipexpert.com
SW1-1(config)# no ip domain-lookup
SW1-1(config)#
Configuring the domain-name and DNS lookups is equal as to how its done within IOS.
To ensure both SSH and Telnet connections are allowed towards the management (mgmt0) interface of
the Nexus you should enable Telnet using the feature command. SSH is the default mechanism to
manage the Nexus switch and therefore the Telnet feature needs to be re-enabled. By default the switch
rejects any attempts of telnet.
To save your configuration using the wr command. The only option is to create a CLI alias, which can be
executed from anywhere in the CLI. The write command is reserved for the erasing of the configuration
as we did at the beginning of the lab, so we need to use an abbreviation for that (what most engineers
use in real-life as well).
SW1-1(config)# sh run int mgmt0
!Command: show running-config interface mgmt0
!Time: Sun Aug 26 19:24:59 2012
version 5.1(5)
interface mgmt0
ip address 10.160.48.241/24
SW1-1(config)# feature telnet
SW1-1(config)# cli alias name write copy run start
Invalid alias name. Reserved word.
SW1-1(config)# cli alias name wr copy run start
SW1-1(config)# wr
[########################################] 100%
Copy complete, now saving to disk (please wait)...
SW1-1(config)#
Note: The management IP address wasnt erased during the erase of the configuration. This doesnt
mean you can just erase your configuration and reload from the Ethernet management connection,
because you are required to set a password for the admin user. Meaning you cant log-in using the SSH
connection that is by default enabled.
By default the SSH log-in is enabled.
24
IPexpert:~ rick$ ssh admin@10.160.48.241

The authenticity of host '10.160.48.241 (10.160.48.241)' can't be
established.
RSA key fingerprint is 95:23:75:04:1c:2e:59:f1:be:a1:e5:f7:ff:62:6d:80.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.160.48.241' (RSA) to the list of known
hosts.
Password:
Last login: Sun Aug 26 19:13:54 2012
Bad terminal type: "xterm-256color". Will assume vt100.
SW1-1# exit
Connection to 10.160.48.241 closed.
RetinaRick:~ rick$ telnet 10.160.48.241
Trying 10.160.48.241...
telnet: connect to address 10.160.48.241: Connection refused
telnet: Unable to connect to remote host
Only after enabling the Telnet feature on the CLI you are able to establish a Telnet connection to the
device.
IPexpert:~ rick$ telnet 10.160.48.241
Trying 10.160.48.241...
Connected to 10.160.48.241.
Escape character is '^]'.
Password:
telnet> q
Connection closed.
IPexpert:~ rick$
Configure the other tasks by just navigating through the configuration. You will never remember all
options available within the OS, but by just navigating using keywords you should be seeing a lot of hints
pointing to the right solution according to what was asked in the task.
SW1-1(config)# cdp format device-id ?
mac-address
Mac-address of the Chassis
serial-number Chassis Serial Number/OUI
system-name
System name/Fully Qualified Domain Name (Default)
SW1-1(config)# cdp format device-id serial-number
SW1-1(config)# cdp advertise v2
SW1-1(config)# int mgmt0
25
SW1-1(config-if)# no cdp enable

SW1-1(config-if)# sh run int mgmt0
!Time: Sun Aug 26 19:56:49 2012
version 5.1(5)
interface mgmt0
no cdp enable
ip address 10.160.48.241/24
SW1-1(config-if)#
both
Set max
input
Set max
output Set max
rate-limit cpu direction ?

input and output packet rate
input packet rate
output packet rate
SW1-1(config-if)# rate-limit cpu direction both pps 999 action log

SW1-1(config-if)# sh run int mgmt0
!Time: Sun Aug 26 19:57:20 2012
version 5.1(5)
interface mgmt0
no cdp enable
ip address 10.160.48.241/24
rate-limit cpu direction both pps 999 action log
Task 2: Implement VLANs

The first task tells you to configure all inter switch links (as seen on the topology drawing) to be
configured as layer 2 trunks with 802.1Q encapsulation. Besides that you should only allow a certain
number of vlans, meaning the allowed vlans command should be used.
SW1-1(config)# default interface e3/1
SW1-1(config)# int e3/1
SW1-1(config-if)# sh run int e3/1
!Command: show running-config interface Ethernet3/1
!Time: Thu Sep 6 20:24:28 2012
version 5.1(5)
interface Ethernet3/1
SW1-1(config-if)# switchport
SW1-1(config-if)# sw mode trunk
SW1-1(config-if)# sw trunk allowed vlan 100-499
This will cause VLANS to be overwritten. Continue anyway? [yes] yes
26
Because this command caused a lot of outages in the world, Cisco implemented a security to ask you if
you are really sure that you want this command to be entered.
!Time: Thu Sep 6 20:24:39 2012
version 5.1(5)
switchport
switchport mode trunk
switchport trunk allowed vlan 100-499
In Classic IOS you only saw the shutdown command under the interface. Within NX-OS the opposite is
true and you only see the no shutdown. Dont think the port is enabled when you see this
configuration. You really need to see the no shutdown command under the interface before it really
works!
SW1-1(config-if)# no shut
!Time: Thu Sep 6 20:24:47 2012
version 5.1(5)
switchport
no shutdown
SW1-1(config-if)#
To remove a specific range (or one VLAN) from the allowed range, you dont need to enter the whole
range again as you would need to in older IOS releases. NX-OS supports the remove command and an
except command.
Be aware when using the except command, because this will remove all VLANs from the trunk without
warning you! Besides its not what the task asked you. You should remove the VLAN from the allowed
range without specifying it. We do this with the remove command.
SW1-1(config-if)# switchport trunk allowed vlan except 333
!Time: Thu Sep 6 20:29:42 2012
27
version 5.1(5)
switchport
switchport trunk allowed vlan 1-332,334-3967,4048-4093
no shutdown
SW1-1(config-if)# sw trunk allowed vlan 100-499
This will cause VLANS to be overwritten. Continue anyway? [yes]
SW1-1(config-if)# sw trunk allowed vlan remove 333
!Time: Thu Sep 6 20:30:05 2012
version 5.1(5)
switchport
switchport trunk allowed vlan 100-332,334-499
no shutdown
SW1-1(config-if)#
Of course the proctor will never be able to verify if you used this single command, but this workbook
prepares you for the lab and every possible question. So this task shows you the possibilities of the OS.
The next tasks will be about configuring the VTP protocol. This protocol ensures all VLANs are
distributed throughout the network. As with almost all features within NX-OS you need to enable it first
before configuring it. The tasks in the workbook will push you to configure about every feature VTP has
and the following configuration will show you all those steps.
If you check the Cisco Documentation about Layer 2 Switching you will see similar configuration
examples which is also what you will see when you sit the lab. Most questions can be related to the
Documentation as that was used during the creation of the lab itself.
SW1-1(config)# feature vtp

Once enabled the VTP command becomes available. When you check the options
you have, you will see about every task can be answered with these
commands.
SW1-1(config)# vtp ?
domain
Set the name of the VTP administrative domain
file
Set the name of the VTP file name
mode
Configure VTP device mode
password Set the password for the VTP administrative domain
pruning
Set the adminstrative domain to permit pruning
28
version
Set the adminstrative domain to VTP version
SW1-1(config)# vtp domain IPexpert

SW1-1(config)# vtp pruning
VTP pruning is the option to allow VLANs to be removed from switches where they are not required.
This limits the broadcast domains created (automatically) using VTP. This is also the main reason why
VTP is not recommended to be used in large datacenters.
SW1-1(config)# vtp version ?
<1-2> Set the adminstrative domain to VTP version
The task asks you to configure the latest version. Which is the highest number?
SW1-1(config)# vtp version 2
SW1-1(config)# vtp file ?
bootflash: URI for vlan.dat
usb1:
URI for vlan.dat
usb2:
URI for vlan.dat
Vlan.dat has historically been the VLAN database. Today its the VTP database which is separately stored
on the flash. When you erase the configuration of a NX-OS device, the VTP information will remain on
the flash and will be configured again when re-enabled.
SW1-1(config)# vtp file ipexpert.dat
SW1-1(config)# vtp mode server
The mode we are looking for in the task is that SW1 is the VTP server for the network, while SW2 and
SW3 are the clients. This way SW2 and SW3 are not able to create vlans. When you try to do this, you
will get an error message.
SW1-1(config)# vtp password ipexpert
Verify that everything is properly configured

SW1-1(config)# sh vtp status
VTP Status Information
---------------------VTP Version
Configuration Revision
Maximum VLANs supported locally
Number of existing VLANs
VTP Operating Mode
VTP Domain Name
VTP Pruning Mode
VTP V2 Mode
VTP Traps Generation
:
:
:
:
:
:
:
:
:
2 (capable)
4
1005
9
Server
IPexpert
Enabled (Operationally Enabled)
Enabled
Disabled
29
MD5 Digest
: 0xF3 0xA6 0x62 0x7A 0xB1 0x85 0x77 0x40
Configuration last modified by 0.0.0.0 at 8-26-12 21:14:51
Local updater ID is 0.0.0.0
VTP version running
: 2
To verify the VTP password a different command is required

SW1-1(config)# show vtp password
VTP password: ipexpert
After creating the VLANs in the next task you should verify on all switches if VTP pushed the information
to those switches. After configuring the names of the VLANs, verify again, as VTP should be pushing that
information towards all switches as well.
SW1-1(config)# vlan
SW1-1(config-vlan)#
SW1-1(config)# vlan
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
101-104
exit
101
name IPexpertVLAN101
vlan 102
vlan 103
vlan 104
sh vlan
VLAN Name
---- -----------------------------------1
default
101 IPexpertVLAN101
102 IPexpertVLAN102
103 IPexpertVLAN103
104 VLAN0104
1002 fddi-default
1003 token-ring-default
1004 fddinet-default
1005 trnet-default
VLAN
---1
101
102
103
104
1002
1003
1004
1005
Type
----enet
enet
enet
enet
enet
enet
enet
enet
enet
Status
Ports
--------- -------------------------active
active
active
active
active
suspended
suspended
suspended
suspended
Eth3/1
Eth3/1
Eth3/1
Eth3/1
Vlan-mode
---------CE
CE
CE
CE
CE
CE
CE
CE
CE
Remote SPAN VLANs
30
-----------------------------------------------------------------------------Primary Secondary
------- --------------
Type
---------------
Ports
-------------------------------------
As you can see the name of VLAN 104 hasnt been changed yet. This is caused by the fact that VLANs are
still located in some sort of database within the configuration. You first need to exit out of the VLAN
configuration before the changes are applied
SW1-1(config-vlan)# exit
SW1-1(config)# sh vlan
VLAN Name
---- -----------------------------------1
default
101 IPexpertVLAN101
102 IPexpertVLAN102
103 IPexpertVLAN103
104 IPexpertVLAN104
1002 fddi-default
1003 token-ring-default
1004 fddinet-default
1005 trnet-default
VLAN
---1
101
102
103
104
1002
1003
1004
1005
Type
----enet
enet
enet
enet
enet
enet
enet
enet
enet
Status
Ports
--------- -------------------------active
active
active
active
active
suspended
suspended
suspended
suspended
Eth3/1
Eth3/1
Eth3/1
Eth3/1
Vlan-mode
---------CE
CE
CE
CE
CE
CE
CE
CE
CE
Remote SPAN VLANs

-----------------------------------------------------------------------------Primary Secondary
------- --------------
Type
---------------
Ports
-------------------------------------
SW1-1(config)#
31
Now verify the VLANs on all 3 switches to ensure that everything has been synchronized using VTP.
The last option to configure in this task is a special case. You really need to pay attention to the fact
what is happening in the show command.
Pay close attention that in the IGMP snooping show command VLAN 105 is shown, while in the show
vlan, you dont see this 105. You probably need to check the Cisco Documentation for this feature, but
its meant for pre-staging VLANs without creating them first.
This means its possible to enable IGMP snooping on a VLAN before its created and applying this
configuration immediately when deployed.
Its a corner-case feature, but Cisco is known to use those particular features in 1 or 2 point tasks within
the CCIE lab. So this could be easy points to get when you know what they are looking for, or you could
easily loose points when you dont and also have no clue where to look for. Again looking at Cisco
Documentation helps a lot in this case and it should be easy to spot this feature in the layer 2
documentation guide.
SW1-1(config)# vlan configuration 105

SW1-1(config-vlan-config)# ?
ip
Configure IP features
no
Negate a command or set its defaults
service-policy Configure service policy for an interface
end
Go to exec mode
exit
Exit from command interpreter
pop
Pop mode from stack or restore from name
push
Push current mode to stack or save it under name
where
Shows the cli context you are in
SW1-1(config-vlan-config)# ip igmp snooping
Another feature to configure is a service-policy in this section. With this policy you can apply QoS
markings or Policing at VLAN level instead of port-level. Be aware that you will mark and police ALL
traffic entering and exciting the VLAN on this switch.
SW1-1(config-vlan-config)# service-policy type qos
After enabling the IGMP snooping this shows up in the show output of IGMP snooping just like the task
required you to. Still the VLAN hasnt been created yet.
SW1-1(config)# exit
SW1-1# wr
[########################################] 100%
SW1-1#
32
Save your configuration before moving on with the next task. Frequently saving you configuration might
save you a lot of trouble when you are configuring features that might cause the box to crash. Or you
reload the box without saving and you lose precious points!
Task 3: Implement Private-VLANs
The assignments in this task are created in such a way that you are building a private VLAN based
network. The assignments are becoming quite clear when you read the full task before starting the
configuration as the assignments individually can be a little difficult to understand, but the entire picture
is much clearer.
Private VLANs are set-up in such a way that traffic is protected between certain ports. There is a Primary
VLAN which only contains the so-called Promiscuous ports. These ports can receive all traffic from all
ports that are associated with this primary VLAN. This means that the client ports are in other VLANs
than the promiscuous port.
The secondary VLANs can be create in 2 ways. The Isolated and Community VLANs are these 2 types
of secondary VLANs. The Isolated version means that all ports that are configured in these VLANs cannot
communicate to each other. As pointed out in the drawing below. Within the layer 2 domain of this
particular VLAN ports will never communicate with each other, but can only talk to the promiscuous
port in the primary VLAN which they are associated with. The second type is the Community VLANs. All
interfaces in these VLANs can communicate to each other within the same Community VLAN. This
means that hosts in different community VLANs cannot communicate to each other. Again these
community VLANs are able to talk to the promiscuous port in the primary VLAN which they are
associated with.
The drawing below illustrates the possible (and impossible) traffic flows.
33
The associations / mappings between primary and secondary VLANs need to be manually configured on
the switch ports. The final step is that its also possible to trunk Private VLANs between different
switches. This means that a promiscuous port doesnt need to be on the same switch as the host ports.
Therefore a whole isolated section of your network can be created on different switches.
The promiscuous port doesnt have to be a fixed Ethernet port. Its also possible to create a VLAN
interface (SVI) which is the promiscuous port for that particular Primary VLAN.
The first assignment point you to create a promiscuous port for an external firewall which will receive all
traffic connected to Primary VLAN 200.
As with all features within the NX-OS platform the private VLAN feature also needs to be activated first,
before any commands are available within the CLI.
SW1-1(config)# feature private-vlan
Then the VLAN needs to be created and the VLAN needs to be made primary. Which we do under the
VLAN configuration.
34
SW1-1(config)# vlan 200

SW1-1(config-vlan)# private-vlan ?
association Configure association
community
Configure the VLAN as
isolated
primary
SW1-1(config-vlan)#
SW1-1(config-vlan)#
Primary Secondary
------- --------------
between private VLANs

community private VLAN
isolated private VLAN
primary private VLAN
private-vlan primary
sh vlan private-vlan
Type
Ports
--------------- -------------------------------------
Still we dont see anything, but this is due to the fact that we need to exit out of the VLAN configuration
before its applied to the switch.
ERROR: Private VLANs can only be configured in VTP transparent/off modes
We receive an error that VTP is not possible to carry private VLAN information. This is true for the
supported versions of VTP on the NX-OS platform. Older Catalyst switches running IOS or even CatOS
also support VTP version 3, which is capable of carrying the private VLAN configuration. This code is
currently not implemented in NX-OS.
SW1-1(config)# vtp mode transparent
SW1-1(config-vlan)# private-vlan primary
SW1-1(config-if)# int e3/19
SW1-1(config-if)# sw mode private-vlan promiscuous
When we change VTP to transparent mode we do see that the changes are applied and that we enabled
a primary VLAN. We dont have to create any mappings yet as that will come in the following
assignments.
SW1-1(config)# sh vlan private-vlan
Primary Secondary Type
------- --------- -------------------200
primary
Ports
------------------------------------E3/19
The next assignment is a combination of enabling isolated ports on the same switch. Before they are
able to communicate to the promiscuous port and the firewall some mappings need to be created.
35

SW1-1(config-vlan)# private-vlan isolated
SW1-1(config-vlan)# private-vlan association 201
Ports
------- --------- --------------- -----------------------------------------200
201
isolated
SW1-1(config)#
SW1-1(config-if)# sw private-vlan ?
association
Private vlan trunk association
host-association Set the private VLAN host association
mapping
Set the private VLAN trunk promiscuous mapping
trunk
Set the private vlan trunking configuration
SW1-1(config-if)# sw private-vlan mapping ?
<1-3967,4048-4093> Primary private VLAN
trunk
Private-vlan trunk promiscuous
SW1-1(config-if)# sw private-vlan mapping 200 ?
<1-3967,4048-4093> Secondary VLAN IDs
add
Add a VLAN to private VLAN list
remove
Remove a VLAN from private VLAN list
SW1-1(config-if)# sw private-vlan mapping 200 201
After the creation of the new isolated VLAN a mapping needs to be made between the primary and
secondary VLANs. This is done under the primary VLAN configuration and under the promiscuous port.
This model of having to create the mapping on multiple places is sometimes a bit difficult to remember
every spot where the mapping needs to be done, but in real life you will see you get a lot of flexibility
with this model and you are able to create almost any topology with this flexibility.
Now the final step is to create the same mapping on the isolated host ports.
SW1-1(config)# int ethernet 3/20-21
SW1-1(config-if-range)# switchport mode private-vlan ?
host
Port mode pvlan host
promiscuous Port mode pvlan promiscuous
trunk
Private-vlan trunk promiscuous
SW1-1(config-if-range)# switchport mode private-vlan host
ERROR: Ethernet3/20-21: requested config change not allowed
36
Dont forget to enable the port on the Nexus 7000 as switchport, as by default the ports are in non
switching mode.
SW1-1(config-if-range)# int e3/20
SW1-1(config-if)# switchport mode private-vlan host
SW1-1(config-if)# switchport private-vlan host-association ?
<1-3967,4048-4093> Primary VLAN ID
SW1-1(config-if)# switchport private-vlan host-association 200 ?
<1-3967,4048-4093> Secondary VLAN ID
SW1-1(config-if)# switchport private-vlan host-association 200 201
SW1-1(config-if)#
SW1-1(config-if)# switchport mode private-vlan host
SW1-1(config-if)# sw priv host-association 200 201
SW1-1(config-if)# exit
SW1-1(config)#
Ports
------- --------- --------------- -----------------------------------------200
201
isolated
Eth3/19, Eth3/20, Eth3/21
After the mapping has been created on all 3 ports (promiscuous and host ports) you will see this
mapping in the show vlan. Here all 3 ports are shown, but this doesnt mean that ports can
communicate to each other. The type of isolate means that this is not possible for the host ports
and only for the promiscuous port.
The following 2 assignments point you in the direction of community VLANs. These VLANs are able to
communicate to each other, but not between different community VLANs and of course they are able to
communicate towards the promiscuous port.
Remember that all traffic on this promiscuous port is untagged. So its not possible to distinguish traffic
between different secondary VLANs on the firewall (virtually connected to port Ethernet 3/19).
SW1-1(config)# vlan
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
SW1-1(config-vlan)#
Primary Secondary
------- -------------200
201
202
203
202
private-vlan community
vlan 203
private-vlan community
vlan 200
private-vlan association add 202-203
sh vlan private
Type
Ports
--------------- ------------------------------------isolated
community
community
Eth3/19, Eth3/20, Eth3/21
37
When the new community VLANs are added, again the mapping needs to be created between the
primary and the secondary VLAN.
The following is that the interfaces need to be assigned to the correct community.
SW1-1(config-vlan)# int
SW1-1(config-if-range)#
e3/22-23
switch
switchport mode private-vlan host
switchport private host 200 202
Next are the second set of ports that are not allowed to communicate to hosts in VLAN 202
SW1-1(config-if-range)# int e3/24-25
SW1-1(config-if-range)# switch
SW1-1(config-if-range)# switchport mode private-vlan host
SW1-1(config-if-range)# switchport private host 200 203
SW1-1(config-if-range)# sh vlan private
Ports
------- --------- --------------- -----------------------------------------200
201
isolated
Eth3/19, Eth3/20, Eth3/21
200
202
community
Eth3/22, Eth3/23
200
203
community
Eth3/24, Eth3/25
And finally dont forget to add the community VLANs to the promiscuous port, otherwise they wont be
able to communicate with the firewall!
Dont forget the add keyword in the command; otherwise you will overwrite all current enabled VLANs
just like it is possible with the trunk allowed vlan command!
SW1-1(config-if-range)# int e3/19
SW1-1(config-if)# sw private mapping 200 add 202-203
SW1-1(config-if)# sh vlan private
------- --------- -------------------200
201
isolated
200
202
community
200
203
community
SW1-1(config-if)#
Ports
------------------------------------Eth3/19, Eth3/20, Eth3/21
Eth3/19, Eth3/22, Eth3/23
Eth3/19, Eth3/24, Eth3/25
The next task requires you to create a different primary VLAN. This primary VLAN will not be configured
towards a physical interface, but will be configured under a layer 3 interface on the core switch (SW1-1).
This is a virtual promiscuous port. Configuration however is basically the same as a physical interface!
SW1-1(config-vlan)# private-vlan primary
SW1-1(config)# interface vlan 204
^
38
Invalid interface format at '^' marker.
On the NX-OS platform its not possible to configure VLAN interfaces (SVIs) right away. Again you need
to enable the feature first!
SW1-1(config)# feature interface-vlan
SW1-1(config-if)# ip address 10.1.10.254/24
SW1-1(config-if)# private-vlan mapping 205
On the SVI the private VLAN mapping to the primary VLAN needs to be configured.
SW1-1(config)# sh int vlan 204 private-vlan mapping
Interface Secondary VLAN
--------- ----------------------------------------------------------------
For the next task another isolated VLAN needs to be created. This VLAN is associated with the special
primary VLAN. After that configuration the mapping is visible on the SVI of VLAN 204.
SW1-1(config-vlan)# private-vlan isolated
SW1-1(config-vlan)# vlan 204
SW1-1(config-vlan)# private-vlan association 205
SW1-1(config)# sh int vlan 204 private-vlan mapping
Interface Secondary VLAN
--------- ---------------------------------------------------------------vlan204
205
The next task requires the use of so-called private VLAN trunk links as the hosts are not connected
directly to the SW1-1, but to SW2. Private-VLAN trunk links can carry traffic for normal VLANs and
private VLANs at the same time.
The configuration can be to transfer primary VLANs to multiple switches or to transfer secondary VLANs
across multiple switches. In this task we configure the first trunk link to pass only secondary VLANs.
SW1-1
SW1-1(config-if)# sw mode private-vlan trunk ?
promiscuous Port mode trunk promiscuous
secondary
Port mode trunk isolated
SW1-1(config-if)# sw mode private-vlan trunk secondary
SW1-1(config-if)# sw private-vlan trunk allowed vlan 205
SW1-1(config-if)# sw privat association trunk 204 205
39
SW2
SW2(config)# feature private-vlan
SW2(config)# vtp mode transparant
SW2(config)# vlan 205
SW2(config-vlan)# private-vlan isolated
SW2(config-vlan)# exit
SW2(config)# int e1/1
SW2(config-if)# sw mode private-vlan trunk secondary
SW2(config-if)# sw private-vlan trunk allowed vlan 205
When configuring private VLAN trunks you dont need to change the normal trunk allowed vlan
command, as the private VLANs on the private-vlan trunk allowed list are automatically
added to the normal VLAN allowed list. On SW2 the same configuration is required to transfer the
secondary VLAN.
As SW2 does not know VLAN 204 (the Primary VLAN) its not necessary to create the association,
because the mapping for this is done on SW1-1.
Next are the hosts that need to be configured in the newly created secondary VLAN on SW2. Remember
that VTP is disabled for this use so you need to manually create the private VLAN on SW2. The host
association does need to be made to ensure the interface is a member of VLAN 205.
SW2(config)# int e1/21-23
SW2(config-if-range)# switchport
SW2(config-if-range)# sw mode private-vlan host
SW2(config-if-range)# sw private-vlan host-association 204 205
SW2(config-if-range)# no shut
Final task is that a second trunk link needs to be enabled to extend the current VLANs. You are required
to use a different trunk link for this use. There are 2 ways to solve this case. The easiest way is to just
extend the secondary VLANs, just like we did in the previous assignment. The other solution is to extend
the primary private VLAN to SW2 and create a new VLAN 201 and 202 to connect those special hosts.
The problem which is created then is that the community VLAN on SW2 is not the same as the
community VLAN on SW1, which means that your connectivity is broken and you are not completing the
requirements of this task successfully.
SW1-1
SW1-1(config-if)# sw mode private-vlan trunk secondary
SW1-1(config-if)# sw private-vlan trunk allowed vlan 201-202
SW2
SW2(config-vlan)# private-vlan isolated
40

SW2(config-vlan)# private-vlan community
SW2(config-if)# sw mode private-vlan trunk secondary
SW2(config-if)# sw private-vlan trunk allowed vlan 201-202
Now you have successfully completed task 3!
Task 4: Implement Rapid Spanning-Tree protocol

The Rapid Per-VLAN Spanning-Tree protocol is by default enabled on NX-OS systems. This
implementation is a Cisco proprietary method of working with Rapid Spanning-Tree (IEEE 802.1w).
Classical spanning-tree is no longer supported. Still Rapid-PVST+ (Ciscos implementation) is supporting a
backwards compatibility with classic STP.
One important difference with classical STP is that Rapid-PVST+ assumes that network (core) links are
configured point-to-point without any hops in between (hubs). This assumption ensures a much quicker
convergence time and is interrupt driven instead of timer driven like in classical STP.
The spanning-tree port type is an important command which you should configure wisely! It can help
you convergence both for host and network links a lot when this is properly configured. When host ports
are configured as Edge ports they are also not able to start a Topology Change (TC) towards the network
which initiates spanning-tree convergence. The flap of an edge interface therefore does not start a
topology change in the network. Keeping the spanning-tree network much more stable!
The first task tells you to configure all host connecting ports should be configured as edge ports so they
arent generating topology changes. Do this by using a range command will ease up your life and save
you precious CCIE lab time.
SW2
SW2(config)# interface e1/10-15
SW2(config-if-range)# spanning-tree port type ?
edge
Consider the interface as edge port (enable portfast)
network Consider the interface as inter-switch link
normal
Consider the interface as normal spanning tree port
41
SW2(config-if-range)# spanning-tree port type edge

Warning: edge port type (portfast) should only be enabled on ports
connected to a single
host. Connecting hubs, concentrators, switches, bridges, etc... to this
interface when edge port type (portfast) is enabled, can cause temporary
bridging loops.
Use with CAUTION
Edge Port Type (Portfast) will be configured in 6 interfaces due to the
range command
but will only have effect when the interfaces are in a non-trunking
mode.
SW2(config-if-range)#
SW3
SW3(config-if-range)# spanning-tree port type edge
The command warns you that it will only have effect when the ports are configures in switchport
mode access plus that spanning-tree portfast is enabled so interfaces will come up fast when hosts
are connected.
The second assignment requires you to configure a root configuration for the spanning-tree instance in
VLAN 101. There are 2 ways to do this, but there is no indication which one needs to be used. Then use
whatever you think is your favorite way to configure this.
The first way is by configuring manual priority values. Choose low values for this, just as a best practice.
There are fixed values that you are able to use for configuring the spanning-tree priority. Dont mind if
you dont reminder these from the top of your head. When you type it in wrong, the CLI will
automatically tell you which values are possible to fill-in.
SW2(config)# spanning-tree vlan 101
ERROR: % Bridge priority must be in
Allowed values are:
0
4096 8192 12288 16384 20480
32768 36864 40960 45056 49152 53248
priority 4093
increments of 4096
24576 28672
57344 61440
SW2(config)# spanning-tree vlan 101 priority 4096

SW2(config)#
The next, and currently preferred, way is by enabling the root functionality by using the dedicated
commands for this. This means that the switch will configure a dedicated priority for both the root and
backup root bridge. The default root priority value is 25476. When there is already a bridge on the
network with a lower priority than this (meaning its the current spanning-tree root bridge) the switch
will choose a priority value of 4096 lower than the current lowest priority value in the network, so
when using this command the switch will always become the root bridge. Of course this is not
42
maintained and when a new switch with a lower value comes online, that will become the root bridge.
This is a single configuration task so at the moment you enter the command, the switch will become the
root.
For the backup root bridge this fixed value is 28672 without a fallback mechanism to ensure its always
the backup root. The switch cant determine the backup root bridge priority value as this is unknown on
the local switch.
The next assignment tells you to ensure that the timers of the Rapid-PVST+ process for VLAN 101. You
can easily configure lower timers for this, but you will you know what the best timer values for this
network are? Well thats quite difficult. Therefore NX-OS has an automatic way of configuring this by
using the special diameter command. This diameter command automatically calculates the best
timer values based on the maximum amount of hops between 2 hosts of the network. As our network
consists of 3 layer 2 switches, so the diameter should be configured on 3.
SW2
SW2(config)# spanning-tree vlan 101 root primary diameter 3
SW3
SW3(config)# spanning-tree vlan 101 root secondary diameter 3
For assignment 4 and 5 of this task its required to read these together, otherwise you might be
configuring the same thing twice, which will cost you time. The task is that SW1 is the root bridge for
VLAN 102. Together with that, any new (unknown) switch may become the root bridge of the network.
Of course you will never know what the pre-configuration of that new switch might look like, but we can
take some assumptions here and ensure that the priority values are as low as possible. The key point in
assignment 5 is that the switch will be configured with default spanning-tree priority values. The default
spanning-tree priority value is 32768, which means we need to configure our network for values lower
than that and ensuring SW1 will be the primary root bridge for VLAN 102.
SW1
SW1-1(config)# spanning-tree vlan 101 priority 8192
SW2
SW3
The next assignment is about steering traffic within the spanning-tree network. This means that we are
going to change spanning-tree costs to alter the layer 2 path through the network and influence which
link is going to be blocked. By default Rapid-PVST+ on NX-OS is using the so called short path-cost
43
method. This means that a 16-bit number is used to determine the cost. You can change this so that a
32-bit number is used to calculate the cost.
Then links can be manually changed what cost is used. On trunk links this can be changed per VLAN. By
default all ports are set to auto. This means that according to the link bandwidth a specific cost is
attached. Depending on if short or long path-cost method is configured.
Bandwidth
Short path-cost method
Long path-cost method
10 Mbps
100
2,000,000
100 Mbps
19
200,000
1 Gigabit Ethernet
20,000
10 Gigabit Ethernet
2,000
The table above shows which cost is given to which links dependent on the configured or negotiated
bandwidth.
With a 16-bit number values of 1 up to 65,535 can be configured. The assignment requires you to
configure numbers of over 100,000. This means that all switches need to be configured with a long
path-cost method.
SW1
SW1-1(config)# spanning-tree pathcost method long
SW2
SW2(config)# spanning-tree pathcost method long
SW3
The next task is the true traffic steering / engineering assignment, where you are asked to ensure that
SW1 is always using SW3 to reach the root bridge of VLAN 101. We just configured SW2 to be the root
bridge of VLAN 101.
44
There are a number of things to take a look at in this position. First is that spanning-tree will always use
the shortest path towards the root bridge as primary path. Meaning that it will elect its root port on that
path. Because we have a loop in our network, there will be links in a blocking state. In the current design
when we take a look at root ports and costs. The links between SW1 and SW3 will become blocking as
the following drawing illustrates.
Now we need to change this behavior as we want all traffic to flow through SW3. This means that the
links between SW1 and SW2 need to be on blocking, as we do need a path to the root switch without
any loops. There are various ways in which you are able to achieve the same goal, but the easiest way in
this case is by raising the cost of the links from SW1 to SW2. All these links are are 10 Gigabit Ethernet
connections so the default cost of all links in the network is 2,000. We are required to change the cost
with values higher than 100,000. So lets change them to 4,000,000 to ensure that no matter what kind
of connection is made between those switches, the default path cost method will never elect it as a
better path than the path between SW1, through SW3 to SW2.
45
The drawing above illustrates the changes and how traffic will flow towards the root bridge. The links
between SW1 and SW2 are now in blocking state to prevent loops in the network and we accomplished
the task!
SW1-1(config)# interface Ethernet3/1-4
SW1-1(config-if-range)# spanning-tree cost 4000000
The next assignment requires you that for all inter-switch-links the fourth link is used as the primary link.
All other links will become blocking. Cost is not an option, because the cost is based on bandwidth and
not on a port level. Spanning-tree knows another config knob to use in the case where there are
multiple equal bandwidth links between bridges like in our network. In normal networks you would use
port-channeling for this features, but in this case we have multiple ports, of which most will become
blocking. Either way we want to ensure that the fourth link will become primary.
This is done using the port-priority feature. As a tie-breaker when default port-priority of 128 is used the
port number is used. So in this case the lowest number wins, which is never the fourth link in our case.
We want to change the port-priority on the fourth link to a lower value than the default of 128. Be
aware that the port-priority can only be configured with increments of 32. Again just like the spanningtree root priority, the CLI will tell you which values are available for use in this case.
We will use priority 64 (any number lower than 128 is good enough) and place it on all 4th links
between the 3 switches in the topology.
SW1
SW1-1(config-if)# spanning-tree port-priority ?
<0-224> Port priority in increments of 32
SW1-1(config-if)# spanning-tree port-priority 100
ERROR: % Port Priority in increments of 32 is required
46
Allowed values are:

0
32
64
96
128
160
192
224

SW1-1(config-if)# interface e3/8
SW1-1(config-if)#
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
interface e1/4
spanning-tree port-priority 64
interface e1/8
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
interface e1/4
interface e1/8
The next assignment asks you to configure the lowest convergence times for VLAN 103. Now we cant
use the diameter command as that will not configure the lowest times. In this case the solution is
simple. Configure all spanning-tree timers to their lowest value possible.
SW1
SW1-1(config)# spanning-tree vlan 103 ?
<CR>
,
Multi range separator
Range separator
forward-time Set the forward delay for the spanning tree
hello-time
Set the hello interval for the spanning tree
max-age
Set the max age interval for the spanning tree
priority
Set the bridge priority for the spanning tree
root
Configure switch as root
SW1-1(config)# spanning-tree vlan 103 hello-time ?
<1-10> Number of seconds between generation of config bpdu
SW1-1(config)# spanning-tree vlan 103 hello-time 1
SW1-1(config)# spanning-tree vlan 103 forward-time ?
<4-30> Number of seconds for the forward delay timer
SW1-1(config)# spanning-tree vlan 103 forward-time 4
SW1-1(config)# spanning-tree vlan 103 max-age ?
<6-40> Maximum number of seconds the information in a bpdu is valid
SW1-1(config)# spanning-tree vlan 103 max-age 6
47
SW2
SW2(config)# spanning-tree vlan 103 hello-time 1
SW2(config)# spanning-tree vlan 103 forward-time 4
SW2(config)# spanning-tree vlan 103 max-age 6
SW3
SW3(config)# spanning-tree vlan 103 hello-time 1
SW3(config)# spanning-tree vlan 103 forward-time 4
SW3(config)# spanning-tree vlan 103 max-age 6
The next assignment is a little bit cryptical. This is mainly done to test your skills of finding examples in
the documentation very fast. When you look up the layer 2 documentation of NX-OS and search for
Rapid Connectivity there is only 1 hit. This is also the technology making Rapid-PVST+ really rapid. As it
will assume a point-to-point link infrastructure instead of a shared infrastructure (multiple layer 2
bridges on a single segment).
The answer is quite simple. Make every inter switch link a point-to-point link.
SW1
SW1-1(config)# int e3/1-8
SW1-1(config-if-range)# spanning-tree link-type point-to-point
SW2
SW2(config-if-range)# spanning-tree link-type point-to-point
SW3
SW3(config-if-range)# spanning-tree link-type point-to-point
The final assignment requires you to clear all spanning-tree configuration from the switches so you can
make a clean slate before moving on to the next section of this chapter
Task 5: Implement Multiple Spanning-Tree protocol

The Multiple Spanning-Tree protocol is much alike the Rapid Spanning-Tree protocol as tested in the
previous task. The main difference lies in the fact that its not VLAN or single instance based, but on
multiple instances where you can configure multiple VLANS to belong to a specific instance or MSTI.
These MSTIs / instances are independent from each other topology wise and should not interfere with
each other once a convergence occurs in the MSTI.
This causes a much higher scale as the control-plane overhead of this protocol is much less than the
Rapid-PVST+ protocol used in the previous task. However it has some downsides as that configuration of
the VLAN to instance mapping needs to be equal on all switches otherwise you will get serious outages
on your network. This requires good thinking about which VLAN is added where.
48
You will see that most assignments in this task are equal to the Rapid-PVST+ task, but require different
thinking, where you need to apply instances instead of VLANs.
First assignment asks you to configure MST (IEEE 802.1s) on all 3 switches. Then you need to configure
some parameters and ensure that the MST is working on all switches. What this means is that all
configuration like name, revision number and VLAN to instance mapping needs to be equal on all
switches in order for MST to work properly. Otherwise you will run into issues in where some traffic
might even be blocked.
SW1
SW1-1(config)# spanning-tree mode mst
SW1-1(config)# spanning-tree mst configuration
SW1-1(config-mst)# ?
abort
Exit region configuration mode, aborting changes
instance
Map vlans to an MST instance
name
Set configuration name
no
private-vlan Set private-vlan synchronization
revision
Set configuration revision number
show
Display region configurations
end
Go to exec mode
exit
pop
push
where
SW1-1(config-mst)#
SW1-1(config-mst)#
SW1-1(config-mst)#
SW1-1(config-mst)#
SW1-1(config-mst)#
name IPexpert
revision 5
instance 1 vlan 10-99
SW2
SW2(config)# spanning-tree mode mst
SW2(config)# spanning-tree mst configuration
SW2(config-mst)# name IPexpert
SW2(config-mst)# revision 5
SW2(config-mst)# instance 1 vlan 10-99
SW3
SW3(config)# spanning-tree mode mst
SW3(config-mst)# name IPexpert
SW3(config-mst)# revision 5
49
The next assignment requires you to use your imagination or check the documentation really quick.
There is a specific functionality under the same spanning-tree mst configuration
configuration knob where you can configure synchronization of private-vlans. This means that no matter
what number the secondary VLANs have, they are always mapped to the instance of the primary VLAN
to ensure an equal layer 2 topology for all private VLANs contained under 1 primary VLAN.
SW1
SW1-1(config)# spanning-tree mst configuration
SW1-1(config-mst)# private-vlan synchronize
SW2
SW2(config-mst)# private-vlan synchronize
SW3
SW3(config-mst)# private-vlan synchronize
Verify that all configurations were done properly. Be aware that accept the VLANs that you configured.
All other VLANs (even when not created) are still mapped to instance 0. Therefore you should always be
aware of that (unconfigured) instance 0 when designing you spanning-tree topology.
SW1-1(config)# sh spanning-tree mst configuration
Name
[IPexpert]
Revision 5
Instances configured 4
Instance Vlans mapped
-------- -------------------------------------------------------------------0
1-9,200-799,1300-4094
1
10-99
2
100-199
3
800-1299
------------------------------------------------------------------------------
The next assignment lets you configure SW2 to be the primary root bridge for instance 1. You should use
the lowest possible value, which is 0.
SW1-1(config)# spanning-tree mst 0 prio 0
SW1-1(config)# sh span mst
##### MST0
Bridge
Root
Regional Root
Operational
Configured
Interface
vlans mapped:
1-9,200-799,1300-4094
address 0050.563b.01b1 priority
32768 (32768 sysid 0)
this switch for the CIST
this switch
hello time 2 , forward delay 15, max age 20, txholdcount 6
hello time 2 , forward delay 15, max age 20, max hops
20
Role Sts Cost
Prio.Nbr Type
50
---------------- ---- --- --------- -------- ------------------------------Eth2/1

Desg FWD 20000
128.257 P2p
##### MST1
Bridge
Root
vlans mapped:
10-99
address 0050.563b.01b1
this switch for MST1
priority
(0 sysid 1)
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Eth2/1
Desg FWD 20000
128.257 P2p
##### MST2
Bridge
Root
vlans mapped:
100-199
this switch for MST2
32770 (32768 sysid 2)
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Eth2/1
Desg FWD 20000
128.257 P2p
From SW2 its clearly visible that SW1-1 is the root bridge for instance 1.
SW2(config)# sh span mst
##### MST0
Bridge
Root
vlans mapped:
1-9,200-799,1300-4094
32768 (32768 sysid
32768 (32768 sysid
port
Eth2/1
path cost
0
Regional Root address 0050.563b.01b1 priority
32768 (32768 sysid
internal cost 20000
rem hops
Operational
hello time 2 , forward delay 15, max age 20, txholdcount
Configured
0)
0)
0)
19
6
20
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Eth2/1
Root FWD 20000
128.257 P2p
##### MST1
Bridge
Root
vlans mapped:
10-99
port
Eth2/1
priority
priority
cost
32769 (32768 sysid 1)

1
(0 sysid 1)
20000
rem hops 19
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Eth2/1
Root FWD 20000
128.257 P2p
##### MST2
vlans mapped:
100-199
51
Bridge
Root
port
Eth2/1
priority
priority
cost
32770 (32768 sysid 2)

32770 (32768 sysid 2)
20000
rem hops 19
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Eth2/1
Root FWD 20000
128.257 P2p
The next assignment asks you to try and configure SW3 to be the root bridge for the MST instance 1
with use of the special root priority command. Why does this fail? Well thats because what was
explained in the Rapid-PVST+ section. The command will (when it sees a lower priority root bridge
currently online) to lower the root priority with 4096 to ensure this switch becomes the root. The
problem is that SW2 is already using the lowest possible value. So this is not possible and the command
will be rejected.
SW3(config-if)# spanning-tree mst 1 root primary
ERROR: Failed to set root bridge for MST 1
% It may be possible to make the bridge root by setting the priority
% for some (or all) of these instances to zero.
SW3(config)#
SW3(config)# sh span mst
##### MST0
Bridge
Root
vlans mapped:
1-9,200-799,1300-4094
address 0050.563b.01c1 priority
32768 (32768 sysid
32768 (32768 sysid
port
Eth2/2
path cost
0
Regional Root address 0050.563b.01b1 priority
32768 (32768 sysid
internal cost 20000
rem hops
Operational
hello time 2 , forward delay 15, max age 20, txholdcount
Configured
Interface
-----------------Eth2/1
Eth2/2
##### MST1
Bridge
Root
##### MST2
Bridge
Root
0)
19
6
20
Role Sts Cost

Prio.Nbr Type
---- --- --------- -------- ----------------------------Altn BLK 20000
Root FWD 20000
vlans mapped:
10-99
address 0050.563b.01c1
port
Eth2/2
Interface
-----------------Eth2/1
Eth2/2
0)
0)
128.257
128.258
P2p
P2p
priority
priority
cost
32769 (32768 sysid 1)

1
(0 sysid 1)
20000
rem hops 19
Role Sts Cost

Prio.Nbr Type
---- --- --------- -------- ----------------------------Altn BLK 20000
Root FWD 20000
128.257
128.258
P2p
P2p
vlans mapped:
100-199
address 0050.563b.01c1 priority
32770 (32768 sysid 2)

32770 (32768 sysid 2)
52
port
Interface
-----------------Eth2/1
Eth2/2
Eth2/2
cost
20000
rem hops 19
Role Sts Cost

Prio.Nbr Type
---- --- --------- -------- ----------------------------Altn BLK 20000
Root FWD 20000
128.257
128.258
P2p
P2p
Now the next task does not allow configuration on the switch itself which needs to be made backup root
bridge. This means we have to use the default configuration of spanning-tree. The default priority value
is 32,768. This means that when we increase the priority value of SW1 we are sure that SW3 will be
elected the new root bridge once SW2 fails.
SW1-1(config)# spanning-tree mst 1 prio 40960
Again we are asked to configure optimal network timers for this topology. The topology is still similar to
the one as used in the Rapid-PVST+ chapter, so we can use the diameter command here again! Pay
attention that changing other instances than instance 0 is possible for the timers or diameter
command, but this is useless as the timers can only be configured once. Timers are always applied for all
instances at the same time. Therefore the diameter command only works when configured for instance
0.
SW2(config)# spanning-tree mst 0 root primary diameter 3
SW2(config-if)# sh span mst detail
##### MST0
Bridge
Root
Regional Root
Operational
Configured
vlans mapped:
1-9,200-799,1300-4094
24576 (24576 sysid 0)
this switch for the CIST
this switch
hello time 2 , forward delay 9 , max age 12, txholdcount 6
hello time 2 , forward delay 9 , max age 12, max hops
20
<output omitted>
Again we are required to use values over 100,000 when using traffic engineering / steering. So this
means that all switches need to be configured with a long path-cost method.
SW1
SW1-1(config)# spanning-tree pathcost method long
SW2
SW3
53
The traffic steering question is similar to the task before, but instead of steering a VLAN, a whole
instance can be steered. The command is slightly different. Again we use a cost of 4,000,000.
SW1-1(config)# interface Ethernet3/1-4
SW1-1(config-if-range)# spanning-tree mst 2 cost 4000000
The next assignment requires a little bit of thinking. Like the previous task we were also asked to
configure port-priority to determine which local interface between switches will be used as the primary
one. Now with MST this is configurable per instance. As we have 4 instances we want to load share
among all interfaces. Configuring the same lower port-priority per instance on different interfaces does
the trick.
SW1
SW1-1(config-if)# spanning-tree
mst 0 port-priority 64
SW2
SW2(config-if)# spanning-tree
54
SW3
The final 2 assignments require a little digging in the documentation again. These are exotic features
that are easily found when you have the documentation open.
The first asks you to configure the spanning-tree max-hops feature.
SW1
SW1-1(config)# spanning-tree mst max-hops 10
SW2
SW2(config)# spanning-tree mst max-hops 10
SW3
SW3(config)# spanning-tree mst max-hops 10
The final in this task is to enable a pre-standard version of MST on a specific interface. Something rarely
seen in production, but its possible in this version of software so its better that you are aware that its
possible.
SW2(config)# interface e1/16
SW2(config)# spanning-tree mst pre-standard
You have successfully completed the MST task!
55
Task 6: Spanning-Tree and UDLD features

Some miscellaneous features remain in the spanning-tree section and on the CCIE Data Center blueprint
they specifically mentioned UDLD, which is why this is incorporated in this chapter.
With these kinds of features its highly recommended to use the documentation. As most of the times
the question, or specific keywords in it, are already mentioned somewhere in the documentation. Then
they are easily spotted and configured in a few minutes.
So opening up the Cisco Documentation for layer 2 features is highly recommended as that makes
searching a lot faster.
The first assignment points you to the direction of spanning-tree port types. As mentioned in the
spanning-tree chapters this is very relevant for your spanning-tree design. When ports are configured
for Edge functionality they will never trigger a topology change in the network. Within the chapter you
were asked to configure certain ports. Its also possible to configure all unconfigured ports for a
dedicated use.
SW3(config)# spanning-tree port type edge default
Next are the special spanning-tree features. The first asks to ensure a port is shut down when BPDUs are
received. This is directly related to the BPDU guard feature.
SW3(config)# spanning-tree bpduguard enable
The next feature filters BPDUs from edge interfaces or trunk interfaces. Be aware when you configure
this on your production network as this might cause bridging loops within your network!
SW3(config)# spanning-tree bpdufilter enable
The next assignment asks you to configure the BPDU filtering on an interface on SW2. The problem is
that we cant configure the command directly under the interface. This means that we will be
configuring the feature globally for every interface (unconfigured). One thing is that we will be
configuring all interfaces, but including Ethernet 1/10 and that means we accomplished this task.
SW2(config)# spanning-tree port type edge bpdufilter default
56
The question 5 relates to the root guard feature. Usually configured on edge ports. Meaning that it will
protect you against that new root bridges are connected to that particular interface. The port will go in
err-disable state when this occurs.
Before the root guard feature works, you first need to re-enable BPDU processing on the interface,
otherwise the superior BPDUs are already caught in the BPDU filter and never processed. The
spanning-tree bpdufilter disable command will not work in this case as.
SW2(config)# no spanning-tree bpdufilter
SW2(config)# spanning-tree guard root
The next task is also a guard feature. Instead of making the port a root port. You can also guard a port
for becoming a designated port. Again we first need to disable the BPDU filter feature, before the loop
guard feature will work.
SW2(config)# no spanning-tree bpdufilter
SW2(config)# spanning-tree guard loop
The next feature is a special case feature where you would always need the CLI help or the Cisco
Documentation. The command disables the Rapid-PVST+ interoperability when MST is enabled.
SW3(config)# spanning-tree mst simulate pvst disable
The final 2 questions in this task point you in the direction UDLD or Unidirectional Link Detection. This is
a Cisco proprietary protocol where it becomes possible to detect a single sided link failure, both on fiber
optic and copper cables. The drawing below illustrates what is meant with this.
57
The protocol can be enabled in 2 ways. Normal or Aggressive mode. The aggressive mode has the
advantage of faster timers and this ensures that traffic is being discarded.
SW3(config-if)# udld ?
^
% Invalid command at '^' marker.
SW3(config-if)# exit
Even this tiny feature needs to be enabled first!

SW3(config)# feature udld
SW3(config-if)# udld ?
aggressive Enable UDLD aggressive mode for interface(s)
disable
Disable UDLD for fiber interface(s)
enable
Enable UDLD for non-fiber interface(s)
SW3(config-if)# udld aggressive
SW3(config-if)#
And you successfully completed this task!
Task 7: Fabric Extenders

The Fabric Extender features allows you to install so-called remote line-cards to either the
Nexus 7000 or the Nexus 5000 series switches. There are various ways to configure them, which will be
tested in other labs of our CCIE Data Center workbook set. The configuration however is always the
same.
The architecture is so that the FEX (short for Fabric EXtender) is configured through the parent switch
which can be either an Nexus 7000 or an Nexus 5000 switch, but not mixed at the same time. Its
possible with Virtual Port Channels (vPC), which is tested in a later chapter, to connect a FEX to 2 parent
switches and still manage them as a remote line card. However this configuration is tested in the vPC
related chapter.
Current we will be configuring the FEX only to SW2 so we get the following architecture in the network.
58
The configuration for the fabric extenders is relatively simple. This task requires you to match the
output. What we can read from the output and the questions is that we need to configure the FEX with
a special name, we need to turn on the beacon LED on the FEX and we need to assign the FEX number of
101 to this particular FEX.
The FEX number is also the linecard number which you will be configuring the host interfaces with. The
FEX number (or chassis ID) needs to be between 100 and 199 and can be pre-staged. So when
connecting certain FEX switches to the Nexus 7000 or 5000 they will automatically get the right FEX
number associated. This is based on the FEX serial number. When no staging is done, the FEX number is
associated to the FEX when its detected on the interface when its enabled.
Whats also shown in the output is that the FEX is connected through port-channel 4. Now portchanneling will be tested in detail in a later chapter. The purpose in this task is to build a port-channel
where the FEX is connected to. You can do this by just configuring 1 interface in a port-channel and let
the FEX connect to that. Which in that case turns out that the FEX is connected to that port-channel.
The configuration for these tasks are:
SW2(config)# feature fex
SW2(config)# fex 101
SW2(config-fex)# beacon
SW2(config-fex)# description "IPexpert Fabric Extender 1"
SW2(config-fex)# exit
SW2(config)#
SW2(config-if)# channel-group 4 mode on
59
SW2(config-if)# no shutdown
SW2(config)#
SW2(config)# interface port-channel 4
SW2(config-if)# switchport mode fex-fabric
SW2(config-if)# fex associate 101
Task 7 is completed
60
Task 8: Misc features

The final task of this chapter will let you configure some miscellaneous features of the NX-OS platforms.
These features are pretty basic features, but will challenge you to read through the whole section first
before starting, otherwise you might end up with re-doing some configuration which will cost you time.
The feature that is asked for in the final bullet requires you to think in a NX-OS centric way. Meaning you
have to use so-called port-profiles to configure these features.
The port-profiles are typically used in the Nexus 1000V where virtual ethernet interfaces are created
once VMs come online, so the configuration is always based on a template. Within the other Nexus
platforms its also possible to configure these templates for any kind of interface, both physical and
virtual. Then you apply this template to one or more physical / virtual interfaces of the kind of portprofile that you made.
This can save you a lot of work in a production environment too. The configuration is straight forward.
The configuration tasks are based on miscellaneous features that can be enabled on interfaces. First is
flowcontrol, which can be activated both in send and receive direction. The straight/cross cable
detection only works on copper interfaces and is called Auto MDIX. Finally the usage statistics on
show interface are configured using the load-interval command. Since NX-OS you can configure
up to 3 counters to be more granular in the averages that are measured.
SW1-1(config)# port-profile type ?
ethernet
Ethernet type
interface-vlan Interface-vlan type
port-channel
Port-channel type
As previously mentioned, its possible to create different types of port-profiles dependent on the type of
interface that you are configuring with it.
SW1-1(config)# port-profile type ethernet IPEXPERT
SW1-1(config-port-prof)# switchport
SW1-1(config-port-prof)# swi mode trunk
SW1-1(config-port-prof)# sw trunk allowed vlan 101-104
SW1-1(config-port-prof)# flowcontrol receive on
SW1-1(config-port-prof)# no mdix auto
Next up is configuring the features under the port profile. The port-profile is configured like a regular
interface with the same options.
The load-interval is different from IOS. When the command is configured without specifying the counter
its configuring the normal IOS counter, where only 1 average delay timer can be configured. In NX-OS
you can configure up to 3 counters to measure different averages. Counter 1 is the default counter and
numbers 2 and 3 are specifically mentioned in the show interface output.
SW1-1(config-port-prof)# load-interval ?
<30-300> Load interval delay in seconds
61
counter
Specify counter for this load interval
SW1-1(config-port-prof)# load-interval counter ?

<1-3> Specify counter for this load interval
SW1-1(config-port-prof)# load-interval counter 1 ?
<30-300> Load interval delay in seconds
SW1-1(config-port-prof)# load-interval counter 1 30
SW1-1(config-port-prof)# state enabled
SW1-1(config-port-prof)# no shut
One important note for port-profiles is that you should never forget the state enabled command.
Without this command the port-profile will never be active or will be enabled, so interfaces will not
come up with the right configuration without this config knob.
SW1-1(config-port-prof)#
SW1-1(config-port-prof)# sh run port-profile
!Command: show running-config port-profile
!Time: Tue Sep 11 02:57:34 2012
version 5.1(5)
port-profile type ethernet IPEXPERT
switchport
flowcontrol receive on
no mdix auto
load-interval counter 1 30
no shutdown
state enabled
SW1-1(config-if-range)# int e5/16-18

SW1-1(config-if-range)# inherit port-profile IPEXPERT
SW1-1(config-if-range)# sh run int e5/16
!Time: Tue Sep 11 02:58:52 2012
version 5.1(5)
inherit port-profile IPEXPERT
62
After applying the port-profile to the interfaces, meaning that the configuration is only visible once in
the configuration, the interface configuration only shows the application of the port-profile to the
interface.
SW1-1(config-if-range)# sh int e5/16
Ethernet5/16 is down (Link not connected)
Hardware: 10/100/1000 Ethernet, address: e05f.b930.3a2b (bia
e05f.b930.3a2b)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
auto-duplex, auto-speed
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is on, output flow-control is off
Auto-mdix is turned off
Switchport monitor is off
EtherType is 0x8100
Last link flapped never
Last clearing of "show interface" counters never
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 1 minute (60 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun
0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
0 interface resets
When verifying the interface. All configuration from the port profile is visible, like the Auto MDIX, the
3 load interval counters and the layer 2 trunk mode.
Now you have successfully completed task 8 and chapter 1 of this workbook. Take a little break before
continuing with configuring the layer 3 routing features!
63
Chapter 3: Data
Center Networking
Layer 3
Infrastructure (NXOS)
Data Center Networking Layer 3 Infrastructure is intended to let you be familiar with the NX-OS Layer 3
features on the Nexus platforms to create a basic routed network. We highly recommend creating your
own diagram at the beginning of each lab so you are able to draw on your own diagram, making it much
easier when you step into the real lab. The lab is divided in two pieces. During the first tasks you will be
configuring a dynamically routed layer 3 network using EIGRP and OSPF protocols. The second part of
this chapter is based on the Cisco proprietary technologies FabricPath and OTV. Multiple topology
drawings are available for this chapter.
64
General Rules
particular chapter
3 hours
65
Solutions
Task 1: Layer 3 topology set-up

The first task is based on drawing 1 as illustrated below. The assignments in this task require you to
configure the network in a very self-supporting way. You get to choose which layer 2 trunks you are
going to configure and you have the responsibility to configure the VLANs in a proper way. Are you going
to do it using VTP or not? Its all up to you, what you think is best to solve this task. During the real
exam, you will find more of these questions where you are free to choose your kind of solution, as long
as it complies with the rules set in the task.
During the initial configuration there are VDCs configured on the Nexus 7000 switch (SW1-1). This
configuration is then used throughout the first couple tasks of this chapter. You can log-in to the VDCs
using the command: switchto vdc <vdcname>. To switch between VDCs use the same command.
To go back to the default VDC you can use: switchback.
SW1-1# switch
switchback
switchto
SW1-1# switchto ?
vdc Manage Virtual Device Context
SW1-1# switchto vdc
SW1-1 VDC number
SW1-2 VDC number
SW1-3 VDC number
SW1-4 VDC number
?
1
2
3
4
66
SW1-1# switchback ?
<CR>
There is not really a best way to configure this network. As long as it works and within the requirements
of the chapter the solution is good. The following configuration is considered best practice in this case.
The first assignment is to ensure that all relevant switches carry VLANs that are required for those
switches. The topology is designed to be built using Switched Virtual Interfaces (SVIs) also called VLAN
interfaces. The SVIs are created as Layer 3 end of the layer 2 domain and are very common in
Datacenter environments. We are using the SVIs in this topology as it saves a lot of physical cables and
ensures a very flexible environment where we can add and remove certain hosts from the interfaces.
Along with the creation of the VLANs and the IP addressing you also need to be sure that the VLANs are
transported between the switches that are used as end-point of the interface.
Within the VDC configuration you are able to see the physical interfaces that are connected between
the different logical switches. On those interfaces a trunk connection carrying the specific VLANs should
be configured. Of course you could enable all VLANs on all interfaces, but this is not helping the topology
as spanning-tree will kick in and block at least one connection for each VLAN somewhere in the topology
where (without any other spanning-tree configuration) you arent sure about which link will be blocked
for this.
Now we decide to only configure all VLANs that also have an SVI configured on the same switch. Along
with that only a single or a few VLAN(s) is/are allowed on the trunk link between the 2 switches that are
configured. Thus we are sure that never a loop exists in the layer 2 network. Which in this case means
that spanning-tree will never block any link in our network and we are running a perfect layer 3 network.
The IP addressing is stated in the drawing and we use the VLAN number as subnet ID and the address
noted with the switches (according to hostname) as the host portion of the address. All subnets are /24
or 255.255.255.0.
Next is to turn this theory into configuration:
SW1-1
SW1-1# conf t
End with CNTL/Z.
As we start with a new configuration we need to enable the SVI feature again.
SW1-1(config)# vlan 112,211,113,122,221
We need to create the layer 2 VLANs that are used on this particular switch.
67
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
ip address 198.18.113.11 255.255.255.0

no shutdown
int vlan 112
ip address 198.18.112.11/24
no shutdown
int vlan 211
ip add 198.18.211.11/24
no shut
int vlan 122
ip add 198.18.122.11/24
no shut
int vlan 221
ip add 198.18.221.11/24
no shut
Connection to SW1-2
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
int Ethernet1/11
switchport
switchport trunk allowed vlan 112,211
no shutdown
Connection to SW1-3
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
int Ethernet1/13
switchport
switchport trunk allowed vlan 113
no shutdown
Connection to SW2
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
int Ethernet1/1
switchport
no shutdown
Now to verify everything is configured properly a show ip interface brief is enough to show you all the
details that we just configured.
SW1-1(config-if)# sh ip int brief
IP Interface Status for VRF "default"(1)
Interface
IP Address
Interface Status
Vlan112
198.18.112.11
protocol-down/link-down/admin-up
Vlan113
198.18.113.11
Vlan122
198.18.122.11
Vlan211
198.18.211.11
Vlan221
198.18.221.11
SW1-1(config-if)#
68
The first switch is configured. Now finishing up by configuring all other layer 2 and layer 3 switches in
the network. Both Nexus 5500 switches contain the Layer 3 module so have the possibility to create a
full layer 3 topology.
SW1-2
SW1-2# conf t
SW1-2(config)# vlan 112,211,124,123,132
SW1-2(config)# int vlan 124
SW1-2(config-if)# ip add 198.18.124.12/24
SW1-2(config-if)# int vlan 112
Connection to SW1-1
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int Ethernet1/14
switchport
no shutdown
Connection to SW1-4
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int Ethernet1/15
switchport
no shutdown
Connection to SW3
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int Ethernet1/5
switchport
no shutdown
69
Verification
Interface
IP Address
Interface Status
Vlan112
198.18.112.12
Vlan124
198.18.124.12
Vlan123
198.18.123.12
Vlan211
198.18.211.12
Vlan132
198.18.132.12
SW1-2(config-if)#
SW1-3
SW1-3# conf t
SW1-3(config)# vlan 113,134
Connection to SW1-1
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
int Ethernet1/16
switchport
no shutdown
Connection to SW1-4
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
int Ethernet1/17
switchport
no shutdown
Verification
Interface
IP Address
Interface Status
Vlan113
198.18.113.13
Vlan134
198.18.134.13
SW1-3(config-if)#
70
SW1-4
SW1-4# conf t
SW1-4(config)# vlan 124,134
Connection to SW1-3
SW1-4(config-if)#
SW1-4(config-if)#
SW1-4(config-if)#
SW1-4(config-if)#
SW1-4(config-if)#
int Ethernet1/19
switchport
no shutdown
Connection to SW1-4
SW1-4(config-if)#
SW1-4(config-if)#
SW1-4(config-if)#
SW1-4(config-if)#
SW1-4(config-if)#
int Ethernet1/18
switchport
no shutdown
Verification
Interface
IP Address
Interface Status
Vlan124
198.18.124.14
Vlan134
198.18.134.14
SW1-4(config-if)#
SW2
SW2# conf t
SW2(config)# feature interface-vlan
SW2(config)# vlan 122,221
SW2(config)# int vlan 122
SW2(config-if)# ip add 198.18.122.2/24
SW2(config-if)# no shut
SW2(config-if)# int vlan 221
71
Connection to SW1-1
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int Ethernet1/1
switchport
no shutdown
Connection to SW3
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int Ethernet1/5
no switchport
no shutdown
int Ethernet1/5.100
encapsulation dot1q 100
ip add 198.18.100.2/24
no shutdown
This will be our only Layer 3 interface in the network without SVI. Still a VLAN is configured for this link.
This is configured using a sub-interface on the physical interface. We first make the interface a layer 3
port and then create the sub-interface while specifying the VLAN number using the encapsulation
command.
Verification
SW2(config-if)# sh ip int brief
Interface
IP Address
Interface Status
Vlan122
198.18.122.2
Vlan221
198.18.221.2
Ethernet1/5.100
198.18.100.2
SW2(config-if)#
SW3
SW3# conf t
SW3(config)# vlan 123,132
SW3(config-if)# int vlan 132
72
Connection to SW1-1
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int Ethernet1/1
switchport
no shutdown
Connection to SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int Ethernet1/5
no switchport
no shutdown
int Ethernet1/5.100
encapsulation dot1q 100
ip add 198.18.100.3/24
no shutdown
Verification
SW3(config-if)# sh ip int brief
Interface
IP Address
Interface Status
Vlan123
198.18.123.3
Vlan132
198.18.132.3
Ethernet1/5.100
198.18.100.3
SW3(config-if)#
The final assignment is to create Loopback interfaces on the routers in the network so they can be
advertised. Loopback interfaces are typically used for managing a device in the network. Now with NXOS this is usually done using the mgmt0 interface on the devices. The best practice is indeed to use this
mgmt0 interface, instead you have a strict requirement to have a redundant management interface.
The mgmt0 interface is always non-redundant, so when this dedicated piece of hardware fails, you cant
manage the switch.
Its highly recommended to do software upgrades through the mgmt0 interface as this interface has a
much higher connection which is not rate-limited to the control-plane. When copying software images
through an in-band management connection (over a Loopback, Layer 3 interface or SVI) you will see
much lower transfer rates as this connection to the control-plane is rate-limited.
In this lab the Loopback interfaces will be used for various purposes and are a good test to see if all
routers are in the routed network. Dont think you already passed a full reachability statement yet,
because you could be missing some crucial links in the routing protocol which means you didnt pass the
full reachability and loose precious points.
Now to finish the task, we create all Loopback interfaces on the switches.
SW1-1
SW1-1(config-if)# int Loopback 0
73
For once, its not necessary to enable this functionality within NX-OS!
After the creation of the Loopback interface we can verify its working properly with a show
interface or again a show ip interface brief.
SW1-1(config-if)# do sh int lo0
loopback0 is up
Hardware: Loopback
Internet Address is 198.18.0.1/32
Encapsulation LOOPBACK
0 packets input 0 bytes
0 multicast frames 0 compressed
0 input errors 0 frame 0 overrun 0 fifo
0 packets output 0 bytes 0 underruns
0 output errors 0 collisions 0 fifo
0 out_carrier_errors
SW1-1(config-if)#
SW1-2
SW1-3
SW1-4
SW2
SW2(config-if)# int Loopback 0
SW3
SW3(config-if)# int Loopback 0
74
Task 2: Static routing

This task is to introduce the static routing features of NX-OS.
The initial assignment asks you to ensure that SW1-3 can ping the loopback address of SW1-4. We can
configure a static route for this. A static route is created using a few keywords:
IP Prefix
Network Mask in dotted decimal or / notation
Next-hop interface
Next-hop IP address
Properties
The network mask can be noted in classical dotted decimal notation (255.255.255.0) like in Classical
IOS or could be using the / notation (x.x.x.x/24) which means the same, but is much easier to
configure and easier to see how large the subnet mask is.
Next can either be directly the next-hop IP address where the router will select the next-hop interface
by itself by looking into the routing table for a connected prefix for this next-hop. On certain platforms
this next-hop can also be recursive, meaning it can point to another routing entry which in that
case points to another routing-entry, until it finds a directly connected prefix which is an
interface connected to the router. It depends very much on the platform how many levels of recursion
are supported. Best practice is to create static routes to point directly to a connected interface.
Instead of a next-hop IP address you can also specify a next-hop interface. Which usually points to a
point-to-point (serial or tunnel) interface which means that traffic that enters, needs to exit out on the
other end, so a next-hop IP address is not necessary. You cant configure an Ethernet interface (physical
or SVI) as the next-hop interface, after that you always need to specify the next-hop interface. This is
because an Ethernet interface is a multi-access network. Meaning that multiple hosts reside on this
interface and the router will then never know where to forward the traffic to. Therefore its always
required to specify this destination IP address along with the interface. Now the router will not do a
look-up in its routing table to find the interface to send it out, as you just specified it.
After these items to configure the static route, a few properties can be configured which will be
discussed later in this chapter.
SW1-1(config)# ip route ?
A.B.C.D
IP prefix in format i.i.i.i
A.B.C.D/LEN IP prefix and network mask length in format x.x.x.x/m
SW1-1(config)# ip route 1.1.1.1 ?
A.B.C.D IP network mask in format m.m.m.m
75
SW1-1(config)# ip route 198.18.0.14/32 ?

A.B.C.D
IP next-hop address in format i.i.i.i
A.B.C.D/LEN
IP next-hop prefix in format i.i.i.i/m
ethernet
Ethernet IEEE 802.3z
loopback
Loopback interface
mgmt
Management interface
null
Null interface
port-channel Port Channel interface
vlan
Vlan interface
SW1-1(config)# ip route 1.1.1.1/32 vlan 111 ?
<CR>
<1-255> Route preference
*Default value is 1
A.B.C.D IP next-hop address in format i.i.i.i
name
Specify name of the next hop
tag
Supply tag value with static route
SW1-1(config)# ip route 1.1.1.1/32 2.2.2.2 ?
<CR>
*Default value is 1
name
tag
The static route pointing to the Loopback interface of SW1-4 on SW1-3:

SW1-3(config)# ip route 198.18.0.14/32 198.18.134.14
The destination prefix is the Loopback interface of SW1-4 and the next-hop address is the directly
connected SVI interface where the next-hop is the host on the other-end (SW1-4).
SW1-3# ping 198.18.0.14
PING 198.18.0.14 (198.18.0.14): 56 data bytes
64 bytes from 198.18.0.14: icmp_seq=0 ttl=255 time=0.759 ms
--- 198.18.0.14 ping statistics --5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.363/0.464/0.759 ms
SW1-3#
Now we verify and it works! Or not?

The second part of the question states that we should be able to ping it from SW1-3s loopback address.
Now if we try that. This doesnt work!
SW1-3# ping 198.18.0.14 source 198.18.0.13
PING 198.18.0.14 (198.18.0.14) from 198.18.0.13: 56 data bytes
Request 0 timed out
Request 1 timed out
76
Request 2 timed out

Request 3 timed out
Request 4 timed out
SW1-3#
Pay attention that in classical IOS you were able to use the interface name as a source in this
command, but within NX-OS you always need to specify the source IP address instead of the
interface.
Why doesnt the ping success? Well thats because SW1-4 has no routing entry for the Loopback address
of SW1-3 in its routing table. Therefore a static route also needs to be configured on SW1-4 for the
return traffic back to SW1-3.
After configuring this static route on SW1-4. Both switches are able to ping each others Loopbacks while
using their own Loopback address as source for the traffic.
SW1-3# ping 198.18.0.14 source 198.18.0.1
SW1-3#
Next up in the second assignment of this task is to configure a static route so that SW1-1 and SW1-2 can
ping each others loopback addresses, but instead of using the directly connected links between those 2
switches they should travel across SW1-3 and SW1-4. This means that we need to point the static routes
over the correct interfaces.
SW1-1
SW1-2
Now the correct static routes are created, but a ping wont succeed.
SW1-1# ping 198.18.0.12 source 198.18.0.11
Request 0 timed out
Request 1 timed out
77
Request 2 timed out

Request 3 timed out
Request 4 timed out
SW1-1#
This is because the switches that are in transit of the path, also need to know the routes back to the
loopback addresses of the switches. This means another 2 static routes need to be created on both
SW1-3 and SW1-4 to ensure traffic going back and forth knows the way to go.
SW1-3
SW1-4
After configuring those additional static routes the ping from both switches succeeds, even while using
the local Loopback interface as the source address.
SW1-1# ping 198.18.0.12 source 198.18.0.11
SW1-1#
SW1-2# ping 198.18.0.11 source 198.18.0.12

SW1-2#
The path between the switches can be checked via a Traceroute

SW1-1(config)# traceroute 198.18.0.12 sour 198.18.0.11
traceroute to 198.18.0.12 (198.18.0.12) from 198.18.0.11, 30 hops max, 40
byte packets
78
1 198.18.113.13 (198.18.113.13) 0.671 ms 0.505 ms 0.631 ms

2 198.18.134.14 (198.18.134.14) 1.003 ms 0.8 ms 0.905 ms
3 198.18.124.12 (198.18. 124.12) 0.903 ms 0.6 ms 0.542 ms
SW1-1(config)#
The next assignment is to configure some properties along with the static route. Besides that we need to
configure a black hole for a specific subnet. This means that we point the static route not towards an
interface or to a next-hop IP address, but towards a special interface. This interface is called the null0
interface. In other words: bit bucket. Every packet that is sent to that interface will be dropped and not
processed further.
Besides pointing the static towards the bit-bucket it also needs to have some properties attached, which
are easy to spot with use of the question mark in the CLI.
The only tricky part is that you need to know the default route preference or administrative distance on
NX-OS of static routes as you need to increase it with +1.
SW1-2(config)# ip route 192.0.1.0/24 null 0 ?
<CR>
*Default value is 1
name
tag
After typing the question mark menu, you can already see the default route preference value (in
classical IOS the administrative distance). This should then be upped with 1 to comply with the question.
Now its possible to configure a name for the static route, but for some reason its then impossible to
change the route preference again. There is no specific reason why this isnt possible, this is a limitation
in the current CLI. The name feature has just been released, so the CLI is not totally good with it. These
are things you will see more common in NX-OS, where new features get introduced very frequently. The
OS is still growing in features, where Classical IOS has grown for the last 20 years, therefore some
features are not mature yet.
SW1-2(config)# ip route 192.0.1.0/24 null 0 2 name BLACKHOLE ?
^
% Invalid number at '^' marker.
SW1-2(config)# ip route 192.0.1.0/24 null 0 2 tag 666
After configuring the right static route it can be verified using a show ip route where its shown that the
route preference is now 2 instead of 1 and that the static route is tagged with a value of 666. Route
tagging is used during redistribution to recognize certain routes in your topology as those tags are also
preserved through dynamic routing protocols.
SW1-2(config)# sh ip route
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
79
'[x/y]' denotes [preference/metric]

1.2.3.4/32, ubest/mbest: 1/0
*via Null0, [1/0], 1w0d, static
192.0.1.0/24, ubest/mbest: 1/0
*via Null0, [2/0], 00:00:02, static, tag 666
198.18.0.1/32, ubest/mbest: 2/0, attached
*via 198.18.0.1, Lo0, [0/0], 16:23:24, local
*via 198.18.0.1, Lo0, [0/0], 16:23:24, direct
*via 198.18.0.14, Lo1, [0/0], 15:36:01, local
*via 198.18.0.14, Lo1, [0/0], 15:36:01, direct
Verify the configuration using a show command. Pay attention that you need to use quotes in NX-OS
while doing a search with multiple words.
SW1-2(config)# sh run | in "ip route"
ip route 0.0.0.0/0 10.160.48.1
ip route 1.2.3.4/32 Null0
ip route 192.0.1.0/24 Null0 tag 666 2
ip route 198.18.0.14/32 198.18.134.14
Now by default traffic that is dropped by the switch will send ICMP unreachable messages back to the
sender. These messages can be used by hackers to detect whether a certain destination is reachable or
not. In Classical IOS it was possible to configure this on the Null0 interface to generally disable it for the
dropped traffic through the Null0 interface, but in NX-OS this is only possible to configure on Layer 3
routed interfaces. Besides that a difference is made for ICMP unreachables and port-unreachables. The
question clearly states that the interfaces should not send out any unreachable message, so dont forget
to add this one as well. SW1-2 has 5 VLAN interfaces (SVIs) so they should all be configured to not send
out the unreachable messages.
SW1-2
SW1-2(config-if)# no ip unreachables
SW1-2(config-if)# no ip port-unreachables
80
To finish the task remove all previously configured static routes from the configuration and continue
with the next task.
SW1-1
SW1-1(config)# no ip route 198.18.0.12/32 198.18.113.13
SW1-2
SW1-2(config)# no ip route 192.0.1.0/24 null 0
SW1-3
SW1-4
Task 3: EIGRP
The next task is about configuring the Enhanced Interior Gateway Routing Protocol or EIGRP. The EIGRP
protocol is the first dynamic routing protocol that is tested in the CCIE Data Center. The next task
regarding Open Shortest Path First (OSPF) is the second routing protocol. Other dynamic routing
protocols like RIP, IS-IS and BGP are not tested in this Data Center exam. Usually you will not find BGP
configuration in a typical Data Center environment on the Nexus switches, but that will be the task of
some kind of PE (Provider Edge) router that is above the core routing layer. This exam doesnt focus on
that part and is only interested in the most widely used Interior Gateway Protocol like EIGRP and OSPF.
RIP is an outdated protocol which is used only in very special cases, whilst IS-IS is more a Service
Provider protocol, due to scaling advantages.
The EIGRP protocol is a Cisco proprietary protocol. Officially its an open document where its based on,
so every routing vendor should be able to write its own implementation, except nobody has. Therefore
EIGRP is considered Cisco proprietary.
The EIGRP protocol is a combination of Distance-Vector and Link-State protocols. Its a very fast
converging protocol due to incremental updates sent to neighbors.
81
The DUAL (Diffusing Update Algorithm) calculates the best routes according to the shortest distance
(based on various link properties). When a successor route is chosen this distance combined with the
local link cost is then used as the Feasible Distance. This is the mechanism that makes EIGRP such a fast
converging protocol.
The DUAL does a second run through the routing table and calculates a second best path based on the
Feasible Distance. A route can become second best when that distance (advertised distance) is lower or
equal to the feasible distance. When this would not be the case and a second best path could be any
cost, you could run into routing loops as the router doesnt know the exact topology like a link-state
protocol does, so the traffic could be crossing the same router again. A Feasible Successor is selected
and installed as a backup route to the Successor route. When the primary path fails the Feasible
Successor route is immediately installed as new best path and the router doesnt have to calculate
another path.
Currently with initiatives like IP FastReRoute (FRR) or IP Loop-Free-Alternate (LFA) this mechanism is also
possible on OSPF and IS-IS, but EIGRP has been using this mechanism for a long time and is known to be
very successful in fast convergence!
The EIGRP metric / distance calculation is quite difficult. The full formula is as follows:
metric = [k1 x bandwidth + (k2 x bandwidth)/(256 - load) + k3 x delay
+ k6 x extended attributes] x [k5/(reliability + k4)]
This calculation is not used in any network at this moment. This is because most of the K-values area at 0
and multiplication to 0 always result in a 0. Currently only the values Bandwidth and Delay (K1 and
K3) are used to calculate the metric of an interface in EIGRP.
With the introduction of Wide Metrics (64-bit metrics) additional K-values came into play. The first is
Jitter (in microseconds) and the second is Energy (in watts per kilobit). With this case you can enable
some sort of Performance Routing by using EIGRP metrics. A lower powered device should attract more
traffic than a high powered device for example.
There are a lot of scenarios in which this would be useful. By default the new K6 values are turned off in
the metric calculation. Though the values are added in the EIGRP update packets.
Wide metrics are also backwards compatible with regular metrics. When a non-wide-metric router
receives an update where the extended attributes are included it will transparently pass it through
without updating it, as a wide-metric-enabled router would do. Although it might result in suboptimal
routing in some cases.
One big advantage of EIGRP on NX-OS is that the auto-summary (classful routing) feature has been
disabled by default and is not possible to turn on as well.
The initial configuration of this task is to configure EIGRP on SW1-2 and SW1-4 whilst enabling loopbacks
in this network as well without forming adjacencies.
82
EIGRP uses an autonomous system number to identify if the routers will allow a neighbor relationship
with each other.
Lets configure EIGRP in our Data Center network:

SW1-4(config-if)# feature eigrp
LAN_ENTERPRISE_SERVICES_PKG license not installed. eigrp feature will be
shutdown after grace period of approximately 119 day(s)
We first enable the EIGRP feature, which automatically enables our LAN_ENTERPRISE_SERVICES license
on the box. This particular test router has no license installed, so the CLI warns you about the grace
period which after the feature will be turned off. For testing purposes we dont have anything to
worry about for now.
SW1-4(config)# router eigrp IPEXPERT
The only thing we need to configure for the process is to create the process with a name. We use
IPEXPERT in this question. The reason is that when the AS number is specified as the process name it
automatically gets this AS number assigned and the process is running.
Currently without specifying the AS number, the process is in a shutdown state as we can see in the
verification.
SW1-4(config-router)# sh ip eigrp
IP-EIGRP AS 0 ID 198.18.0.14 VRF default
Process-tag: IPEXPERT
Status: shutdown
Authentication mode: none
Authentication key-chain: none
Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0
IP proto: 88 Multicast group: 224.0.0.10
Int distance: 90 Ext distance: 170
Max paths: 8
Number of EIGRP interfaces: 0 (0 loopbacks)
Number of EIGRP passive interfaces: 0
Number of EIGRP peers: 0
Graceful-Restart: Enabled
Stub-Routing: Disabled
NSF converge time limit/expiries: 120/0
NSF route-hold time limit/expiries: 240/0
NSF signal time limit/expiries: 20/0
Redistributed max-prefix: Disabled
SW1-4(config-router)# autonomous-system 64999
83
To get the process enabled we need to configure the AS number manually using the dedicated
command for this. Now when we verify, the process is running with the correct name and AS number
attached.
SW1-4(config-router)# sh ip eigrp
Process-tag: IPEXPERT
Status: running
Max paths: 8
SW1-4(config-router)#
The rest of the configuration is done under the interfaces that will be placed in the EIGRP process.
Within Classical IOS much more configuration was done under the process, but this has now moved to
the interface.
SW1-4
SW1-4(config-router)# int vlan124
SW1-4(config-if)# ip router eigrp IPEXPERT
SW1-2
SW1-2(config)# feature eigrp
SW1-2(config-router)# exit
SW1-2(config)# int vlan124
SW1-2(config-if)#
Once we enable the EIGRP process on the correct interface, the neighbor adjacency almost immediately
comes up and is working correctly.
SW1-4
SW1-4(config-if)# sh ip eigrp nei
IP-EIGRP neighbors for process 64999 VRF default
84
Address
Seq
Interface
Hold
Uptime
(sec)
Num
0
198.18.124.12
2
SW1-4(config-if)#
Vlan124
12
SRTT
RTO
(ms)
00:00:06
Q
Cnt
200
SW1-2
H
Address
Interface
Hold Uptime SRTT
Seq
(sec)
(ms)
Num
0
198.18.124.14
Vlan124
12
00:00:16 10
2
SW1-2(config-if)#
RTO
Q
Cnt
200
Once we enable the EIGRP process on the correct interface, the neighbor adjacency almost immediately
comes up and is working correctly. Now without any advertised interfaces as its only enabled on a
point-to-point link.
Before we will advertise any subnets we first need to ensure that the adjacency between the
switches/routers is secure as stated in the question.
This is done using the authentication configuration keywords and a key chain with a valid key. Its
possible to configure multiple keys with lifetimes so you can change your EIGRP authentication keys
without downtime by making the acceptance lifetime a little longer than the sending lifetime. That way
new keys are automatically used, but old keys are still accepted for a longer period so other routers get
time to also send this new key.
SW1-4
SW1-4(config)# key chain EIGRP_CHAIN
SW1-4(config-keychain)# key 1
SW1-4(config-keychain-key)# key-string IPEXPERT
As by the default parameters stated at the beginning of this workbook we will configure a password of
IPEXPERT to secure the adjacency.
SW1-4(config-keychain-key)# int vlan124
SW1-4(config-if)# ip authentication ?
key-chain Key-chain
mode
Mode
SW1-4(config-if)# ip authentication key-chain eigrp ?
WORD Process tag (Max Size 20)
85
SW1-4(config-if)# ip authentication key-chain eigrp IPEXPERT ?

WORD Name of key-chain
SW1-4(config-if)# ip authentication key-chain eigrp IPEXPERT EIGRP_CHAIN
SW1-4(config-if)# ip authentication mode eigrp IPEXPERT ?
md5 Keyed message digest
SW1-4(config-if)# ip authentication mode eigrp IPEXPERT md5
The only possible authentication mode is md5. Simple-text / Clear-text authentication is not supported
in EIGRP.
SW1-4(config-if)# sh run int vlan124
!Command: show running-config interface Vlan124
version 5.0(3)
interface Vlan124
ip address 198.18.124.14/24
ip router eigrp IPEXPERT
ip authentication mode eigrp IPEXPERT md5
ip authentication key-chain eigrp IPEXPERT IPEXPERT
no shutdown
SW1-4(config-if)#
After all settings are configured we clearly see that the neighbor has gone down with the following
message:
SW1-4
2012 Sep 19 14:57:39 SW1-4 %EIGRP-5-NBRCHANGE_DUAL: eigrp-IPEXPERT
[14861] (def
ault-base) IP-EIGRP(0) 64999: Neighbor 198.18.124.12 (Vlan124) is down:
auth
entication mode changed
On the other switch the neighbor has also gone down with another message:
SW1-2
2012 Sep 19 14:57:39 SW1-2 %EIGRP-5-NBRCHANGE_DUAL: eigrp-IPEXPERT
[14789] (def
ault-base) IP-EIGRP(0) 64999: Neighbor 198.18.124.14 (Ethernet2/2) is
down: Auth
failure
Now we configure the other switch with the authentication settings to have both ends configured equal.
SW1-2
SW1-2(config)# key chain EIGRP_CHAIN
SW1-2(config-keychain-key)# key-string IPEXPERT
86
SW1-2(config-keychain-key)# int vlan124

SW1-2(config-if)# ip authentication key-chain eigrp IPEXPERT EIGRP_CHAIN
SW1-2(config-if)# ip authentication mode eigrp IPEXPERT md5
SW1-2(config-if)#
After configuring the correct authentication settings the neighbor immediately comes back up and is
working as before:
SW1-2
H
Address
Interface
Hold Uptime SRTT
Seq
(sec)
(ms)
Num
0
198.18.124.14
Vlan124
14
00:00:05 28
6
SW1-2(config-if)#
RTO
Q
Cnt
200
SW1-4
H
Address
Interface
Hold Uptime SRTT
Seq
(sec)
(ms)
Num
0
198.18.124.12
Vlan124
11
00:00:37 7
6
SW1-4(config-if)#
RTO
Q
Cnt
200
Besides the securing of the adjacency we need to enable EIGRP on the Loopback interfaces so they can
be reached from both switches and no attempts to make adjacencies should be done on the Loopback
interfaces. This is done using the passive-interface command.
SW1-2
SW1-2(config)# int lo0
SW1-2(config-if)# ip passive-interface eigrp IPEXPERT
SW1-4
SW1-4(config)# int lo0
SW1-4(config-if)# ip passive-interface eigrp IPEXPERT
Now we are able to ping the loopback interfaces that are dynamically learned on SW1-2 and SW1-4.
SW1-2(config)# show ip route eigrp
87

198.18.0.14/32, ubest/mbest: 1/0
*via 198.18.124.14, Vlan124, [90/130816], 00:03:47, eigrp-IPEXPERT,
internal
SW1-2(config)#
SW1-2(config)#
SW1-2(config)# ping 198.18.0.14 sour 198.18.0.12
SW1-2(config)#
The next step is to configure static routes on SW1-4 and ensure that they are advertised as a summary
route towards SW1-2. We cannot overlap any other networks. With the statics we need to configure this
is done quite easily.
When we configure the routes:
198.18.4.0/24
198.18.5.0/24
198.18.6.0/24
198.18.7.0/24
We see that the third octet is changed.
In binary the third octet looks like this:
00000100
00000101
00000110
00000111
Anything between the last 2 bits is changed in these static routes. So when we create a static route with
a mask that is 2 bits shorter than the /24 meaning a /22 we are good to go!
88
First we configure the static routes on SW1-4 pointing to null0 to ensure they are installed in the routing
table. When you create a static with a next-hop address which is not valid you will not see them being
installed in the routing table, except when they are marked with a permanent option, but this is not
available in NX-OS.
SW1-4(config)#
SW1-4(config)#
SW1-4(config)#
SW1-4(config)#
ip
ip
ip
ip
route
route
route
route
198.18.4.0/24
198.18.5.0/24
198.18.6.0/24
198.18.7.0/24
null0
null0
null0
null0
Next we configure the summary on the interface towards SW1-2 to ensure the statics are summarized
as one entry. Note that summary addresses get a default route preference of 5 in the routing
table.
SW1-4(config-router)# int Vlan124
SW1-4(config-if)# ip summary-address eigrp IPEXPERT 198.18.0.0/22 ?
<CR>
<1-255>
Administrative distance
*Default value is 5
leak-map Allow dynamic prefixes based on the leak-map
SW1-4(config-if)# ip summary-address eigrp IPEXPERT 198.18.0.0/22
Now when we check SW1-2 we still dont see the summary address or any of the static routes appear in
the routing table.
SW1-2(config)# show ip route eigrp
198.18.0.14/32, ubest/mbest: 1/0
internal
SW1-2(config)#
We first need to ensure that the prefixes are injected in the EIGRP topology database by using a process
called Redistribution. This process ensures that prefixes from another protocol (or connected/static
routes) are injected the protocol where its configured. This can cause serious routing loops in the
network so beware when configuring this. As a pre-caution Cisco implemented a failure check that
redistribution always needs to be configured using a route-map to select which prefixes are entering the
redistribution process or not.
In our case we only need to configure these 4 static routes so additional filtering on the route-map is not
strictly necessary though highly recommended when this would be a real test. During the real test you
might find that a later task asks you to configure an additional static route which will then automatically
be redistributed in a protocol, which you probably dont want. This might cause you loosing points in
both that new task and the older EIGRP task which you finished in the morning.
89
Configuring the redistribution enables the static routes to appear in the

EIGRP topology database.
SW1-4(config)# route-map STATIC->EIGRP permit 10
SW1-4(config-route-map)# exit
SW1-4(config-router)# redistribute static ?
route-map Route-map to constrain redistribution
A route-map is required when configuring. You can not apply this command without specifying it
anymore.
SW1-4(config-router)# redistribute static route-map STATIC->EIGRP
Now when checking the routing table on SW1-2 we see the new summary address configured.
SW1-2(config)# sh ip route eigrp
198.18.0.14/32, ubest/mbest: 1/0
internal
198.18.4.0/22, ubest/mbest: 1/0
internal
SW1-2(config)#
In the topology database of SW1-4 we see all 5 prefixes enabled, but only 1 is advertised to the other
routers. This is the advantage of distance-vector style protocols where you send your own routing table
to the neighbor and can therefore easily filter out certain prefixes that you want to hide.
SW1-4(config-if)# sh ip eigrp topology

IP-EIGRP Topology Table for AS(64999)/ID(198.18.0.14) VRF default
Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status
P 198.18.124.0/24, 1 successors, FD is 2816
via Connected, Ethernet2/2
via Summary (51200/0), Null0
via Rstatic (51200/0)
via 198.18.124.12 (130816/128320), Ethernet2/2
via Connected, loopback0
90

SW1-4(config-if)#
Now we dont comply with the question yet as we also want one specific prefix to be leaked through
the summary. This is where we are going to use a leak-map to specify this single prefix to be leaked to
the other router (SW1-2).
SW1-4(config-if)# ip prefix-list EIGRP permit 198.18.5.0/24
SW1-4(config)# route-map LEAK permit 10
SW1-4(config-route-map)# match ip add prefix-list EIGRP
SW1-4(config-if)# ip summary-address eigrp IPEXPERT 198.18.4.0/22 leak-map
LEAK
SW1-4(config-if)#
Now after applying the leak-map we see the leaked prefix also in the routing table of SW1-2.
SW1-2(config)# sh ip route eigrp
198.18.0.14/32, ubest/mbest: 1/0
internal
198.18.4.0/22, ubest/mbest: 1/0
internal
198.18.5.0/24, ubest/mbest: 1/0
external
SW1-2(config)#
The rest of the questions in this task relate to certain EIGRP features. Again opening up the Unicast
Routing Configuration Guide really helps when solving these tasks as most of the question can relate to
something in this guide and walk you thru certain solutions.
The first is to enable wide metrics. Wide metrics are explained in the beginning of this task solution and
enable 64-bit wide metrics instead of 32-bit. These can be scaled down to 32-bit numbers again using a
scaling-factor. This factor can be adjusted along your preference which is also what the question states.
Be sure to enable it on both EIGRP routers.
91
SW1-2
SW1-2(config-if)# router eigrp IPEXPERT
SW1-2(config-router)# metrics version 64bit
SW1-2(config-router)# metrics rib-scale 64
SW1-4
SW1-4(config-router)# metrics version 64bit
SW1-4(config-router)# metrics rib-scale 64
Next up is changing the EIGRP bandwidth utilization on an interface. This value can be changed. The
default is 50% of control-plane traffic, so lowering this with 10% brings it down to 40%.
SW1-2
SW1-2(config-if)# interface Vlan124
SW1-2(config-if)# ip bandwidth-percent eigrp IPEXPERT 40
SW1-4
SW1-4(config-if)# ip bandwidth-percent eigrp IPEXPERT 40
Tuning the timers on an interface affects the neighborship. So beware of a flapping neighbor when
timers are changed. This is because the holddown timer is negotiated in the hello packets of EIGRP. The
question states to change the holddown timer to 4 times a hello packet.
By default hello packets are sent every 5 seconds and by default the holddown timer is set to 15
seconds, meaning it can miss 3 hellos before declaring a neighbor down. So when we change SW1-2 to
a holddown timer of 20 seconds it should negotiate it down towards SW1-4 and from then on use the
new timer
SW1-2(config-if)# ip hold-time eigrp IPEXPERT ?
<1-65535> Seconds before neighbor is considered down
*Default value is 15
SW1-2(config-if)# ip hold-time eigrp IPEXPERT 20
Now we can verify this by issuing a show ip eigrp neighbor detail on the neighboring switch to verify its
using the new holddown timer.
SW1-4(config-if)# sh ip eigrp nei det
H
Address
Interface
Hold Uptime SRTT
Seq
(sec)
(ms)
Num
0
198.18.124.12
Eth2/2
18
00:00:01 39
31
Version 12.4/1.2, Retrans: 0, Retries: 0, Prefixes: 1
RTO
Q
Cnt
234
92
SW1-4(config-if)#
Note that SW1-2 and SW1-4 are now using different values to declare each other down. This is exactly
what the question indicates.
The next 2 questions indicate some tuning to the EIGRP election process. The configuration guide is your
friend while answering these types of questions. The Stuck in Active timer and the Max-Hops
property needs to be changed to finish these questions and are easily found in the Tuning EIGRP
section of the configuration guide.
SW1-2
SW1-2(config-router)# timers active-time 5
SW1-2(config-router)# metric max-hops 50
SW1-4
SW1-4(config-router)# timers active-time 5
SW1-4(config-router)# metric max-hops 50
The final question of this task requires you to change value K3 on the interfaces. This value is used for
calculating the interface delay in the EIGRP metric formula:
metric = [k1 x bandwidth + (k2 x bandwidth)/(256 - load) + k3 x delay
+ k6 x extended attributes] * [k5/(reliability + k4)]
The delay value is an administrative value on the interface, just like bandwidth and can be
administratively changed to do traffic engineering in EIGRP which is more recommended than to
changing the bandwidth value.
SW1-2
SW1-2(config-if)# delay 500
SW1-4
SW1-4(config-if)# delay 500
After changing this on both sides of the link task 3 is finished!
93
Task 4: OSPF
The next task is about configuring the Open Shortest Path First or OSPF. The OSPF protocol is considered
the most widely used dynamic routing protocol. Its an IETF standard and every networking hardware
vendor has its own implementation of this protocol.
The current mostly used version and the version tested on this CCIE lab exam is OSPF version 2. This
version only support IP version 4 and has been out for many years. Currently there is a new version of
OSPF maturing which is Version 3. This version was initially developed solely for IP version 6, but has
now been extended so it also supports IP version 4. In the near future its expected that version 3 will
take over from version 2. The 2 versions are in no way compatible with each other, so you will have to
migrate your whole network over from version 2 to version 3. Some key features have changed in the
new version, particularly how interfaces/links are treated in the protocol.
This workbook focuses on the version used in the CCIE Data Center lab exam, which is OSPF version 2 for
IP version 4.
The OSPF protocol is a Link-State routing protocol. Which means that instead of pushing the routing
table to neighboring routers. The protocol will inform all neighboring routers within a single area of the
state of each connecting link using link-state-advertisements (LSAs). This information is flooded
throughout the OSPF area to all routers. Once all information is gathered the protocol will start a
algorithm known as the Dijkstra algorithm or the Shortest Path First algorithm which is also used in GPS
navigation systems to determine the shortest route/path to any of the links/prefixes that has been
received. To do this, the router places itself as the top of the tree and then branches out to all other
links through routers to determine the fastest route.
The awareness of the entire topology for each router does impact on resources, but with the current
control-plane architectures of modern routers this is never a problem. OSPF networks can now grow up
to tens or hundreds of routers in a single area.
The behavior of OSPF changes as soon as the border of an area is reached. Information between areas is
exchanged in a way more like distance vector routing protocols do this. Information about links in the
area is summarized and then sent to the so-called backbone area (area 0). The behavior to connect
every area to the backbone area is done to prevent loops from occurring in the network. Its now always
known that every next-hop area is area 0 of which then the decision is made where to move on next.
This means that area 0 contains all information about the OSPF network. Its impossible to filter out any
information (except for summarization and non-OSPF external routes) from area 0. The other areas may
have limited visibility on the network. This is mostly done for a more lightweight approach to certain
areas. It means that you can connect a very small router to a highly advances OSPF backbone network
and still be able to participate in the OSPF as this lightweight router in its own area will only get very
limited information regarding the topology (usually a default route).
94
Information is sent through LSA messages. For the CCIE exam its highly recommended to know all the
different types of messages and the numbers attached to that. You have a high chance of running into
questions stating these LSA numbers, where you should know what information is carried within them
and when they are propagated in which kind of area.
The LSA types in OSPF version 2 are:
LSA # Name
Flooding
Description
Router
area
Includes link state information, cost and a list of neighbors.

Every router sends this LSA throughout the area.
Network
area
Sent by the designated router on a multi-access network.

Lists all the routers within the multi-access network.
Summary
network
Contains information regarding each destination in the

area and is sent by the Area Border Router (ABR) towards
area 0. It includes link cost from the ABR towards the
destination (like a distance-vector protocol).
ASBR Summary network
Contains information regarding the Autonomous System

Boundary Router with a cost to reach it. Its sent to area 0
by the ABR of the local area.
AS External
network
Contains information regarding routes external to the OSPF

network. Generated by the ASBR and sent throughout the
entire OSPF network.
Multicast
n/a
Multicast OSPF was never implemented.
NSSA External
area
Same as a type 5, but generated by an ASBR within a Notso-stubby area. These are only flooded within an area and
the ABR converts them to a regular Type 5.
Opaque 1
link
Same as type 10 and 11. Used for expansion to the OSPF

protocol for carrying application information. This type is
mainly used for OSPF Graceful Restart
10
Opaque 2
area
Same as type 9 and 11. Used in MPLS Fast Reroute

scenarios
11
Opaque 3
network
Same as type 9 and 10
95
During the solutions of the task more will be explained about the details regarding the different types of
areas and the different types of network links.
When looking at drawing 2. An immediate issue is spotted when looking at the areas that we need to
configure to comply with this topology. As just discussed, every area should be connected to area 0 to
prevent loops in the network.
Area 264 in this drawing is not connected to area 0 and is therefore not going to work.
There are multiple solutions to this problem, but only one of them is built into NX-OS. OSPF has a built-in
functionality to overcome these situations. In a real-life scenario this will hardly ever occur, because its
much easier to configure an additional interface between 2 routers and create a area 0 connection on
there. Another solution would be to use IP-IP or GRE tunnels between the routers to also ensure an
additional interface between the routers where you could run area 0 on is created.
The built-in solution we are looking for is an OSPF virtual-link. This is also some kind of tunnel
connection between 2 Area Border Routers (ABRs) to create a virtual area 0 connection between a
backbone ABR and a partitioned ABR.
Lets start by creating the OSPF network by building up the adjacencies and advertising the Loopback
interfaces into the OSPF network.
SW1-1(config-if)# feature ospf
96
LAN_ENTERPRISE_SERVICES_PKG license not installed. ospf feature will be

shutdown
after grace period of approximately 119 day(s)
Again the feature needs to be enabled. The Enterprise license is required to run any routing protocol,
including OSPF.
We also need to supply a tag for the OSPF process. In classical IOS this used to be a process number (ID),
but in NX-OS this has changed to a name. In the lab there is no mention of a name, so we use
IPEXPERT, but you can also use 1 just like you would in IOS.
SW1-1(config)# router ospf ?
WORD
Process tag (Max Size 20)
SW1-1(config)# router ospf IPEXPERT
That should be sufficient to enable OSPF on the box. Now we need to enable it on the interfaces.
SW1-1(config-router)# int Vlan113
SW1-1(config-if)# ip router ospf IPEXPERT
SW1-1(config-if)# int Vlan122
SW1-1(config-if)#
SW1-1(config-if)# int lo0
area 2
area 1
area 1
area 0
area 0
area 0
Verifying OSPF is enabled on all interfaces with the correct area mapping:
SW1-1(config-if)# sh ip ospf interface brief
OSPF Process ID IPEXPERT VRF default
Total number of interface: 6
Interface
ID
Area
Status
Lo0
6
0.0.0.0
up
Vlan113
1
0.0.0.2
up
Vlan122
2
0.0.0.1
up
Vlan221
3
0.0.0.1
up
Vlan112
4
0.0.0.0
up
Vlan211
5
0.0.0.0
up
Cost
State
Neighbors
LOOPBACK 0
40
DR
40
DR
40
DR
40
DR
40
DR
97
SW1-1(config-if)#
Now we enable the interfaces on all OSPF routers and see that OSPF adjacencies start to come up.
SW1-2
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int Vlan123
ip router ospf
int Vlan132
ip router ospf
int Vlan112
ip router ospf
int Vlan211
ip router ospf
IPEXPERT area 1
IPEXPERT area 1
IPEXPERT area 0
IPEXPERT area 0
int lo0
ip router ospf IPEXPERT area 0
SW1-3
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
int Vlan113
int Vlan134
ip router ospf IPEXPERT area 0.0.1.8
int lo0
The question states that the dotted decimal notation should be used to configure area 264 in this lab.
This means that we cant just use 265 when configuring it.
When you want to be sure to have the correct notation. You can configure it the normal way and check
a show ip ospf interface brief, as that output always gives you the dotted decimal notation
even when you configured a decimal based area number. Then you can reconfigure it again, as with
grading the proctor will see your decimal based area number instead of the dotted decimal.
SW1-3(config-if)# sh ip ospf int brief
Total number of interface: 3
Interface
ID
Area
Status
Vlan113
1
0.0.0.2
up
Cost
State
40
BDR
Neighbors
1
98
Vlan134
up
Lo0
up
0.0.1.8
40
DR
0.0.0.2
LOOPBACK 0
SW1-3(config-if)#
Now the neighbor has been established, but when we check the routing table. We wont see the
Vlan134 interface in the routing table on SW1-1. This is because the areas are not allowed to be
disconnected from area 0. The solution in this case would, as previously explained, be to create an OSPF
virtual-link to create a virtual connection to area 0 to fix this issue.
SW1-1(config-if)# sh ip ospf nei
Total number of neighbors: 2
Neighbor ID
Pri State
Up Time Address
198.18.0.2
1 FULL/BDR
00:01:05 198.18.122.2
198.18.0.13
1 FULL/DR
00:01:31 198.18.113.13
SW1-1(config-if)# sh ip route
Interface
Vlan122
Vlan113
198.18.0.2/32, ubest/mbest: 1/0

*via 198.18.122.2, Eth2/1, [110/41], 00:01:05, ospf-IPEXPERT, intra
*via 198.18.0.11, Lo0, [0/0], 00:03:55, local
*via 198.18.0.11, Lo0, [0/0], 00:03:55, direct
198.18.0.13/32, ubest/mbest: 1/0
*via 198.18.113.11, Eth2/2, [0/0], 00:03:45, direct
*via 198.18.113.11, Eth2/2, [0/0], 00:03:45, local
*via 198.18.122.11, Eth2/1, [0/0], 00:03:39, direct
*via 198.18.122.11, Eth2/1, [0/0], 00:03:39, local
SW1-1(config-if)#
The virtual-link is created, which results in a new OSPF adjacency. This adjacency is an area 0 adjacency
and across that adjacency the networks area advertised and the communication with area 0 is
established. Within the configuration of the virtual-link its required to specify the area thru which the
virtual-link is transmitting to reach the next area. Whats typically seen is that students configure the
area that needs to be extended as the virtual-link area, but this should be the transit-area. The second
property that needs to be configured is the Router-ID of the Area Border Router (ABR) where the virtuallink terminates.
SW1-3
SW1-3(config-if)# router ospf IPEXPERT
SW1-3(config-router)# area 2 virtual-link 198.18.0.11
SW1-3(config-router-vlink)# exit
99
SW1-1
SW1-1(config-router)# router ospf IPEXPERT
SW1-1(config-router-vlink)# exit
After the virtual-link is created the new VL1 adjacency automatically comes up and the network is now
seen in the OSPF routing table!
SW1-1(config-router)# sh ip ospf nei
Neighbor ID
Pri State
Up Time
198.18.0.13
1 FULL/ 00:00:06
198.18.0.2
1 FULL/BDR
00:08:14
198.18.0.13
1 FULL/DR
00:08:40
SW1-1(config-router)# sh ip route ospf
Address
198.18.113.13
198.18.122.2
198.18.113.13
Interface
VL1
Eth2/1
Eth2/2
198.18.0.2/32, ubest/mbest: 1/0

198.18.0.13/32, ubest/mbest: 1/0
198.18.134.0/24, ubest/mbest: 1/0
*via 198.18.113.13, Eth2/2, [110/80], 00:00:06, ospf-IPEXPERT, inter
Now the rest of the OSPF network needs to be finalized

SW1-4
SW1-4(config-if)# ip router ospf IPEXPERT area 0.0.1.8
SW2
SW2(config-if)# feature ospf
SW2(config)# router ospf IPEXPERT
SW2(config-router)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int Vlan122
int Vlan221
int Ethernet1/5
100
SW2(config-if)#
SW2(config-if)# int lo0
SW2(config-if)# ip router ospf IPEXPERT area 1
SW3
SW3(config-router)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int Vlan123
int Vlan132
int Ethernet1/5
int lo0
After all OSPF configuration has been completed the Loopbacks are reachable throughout the network
without any problems.
SW2(config)# ping 198.18.0.13 source 198.18.0.2
SW2(config)# traceroute 198.18.0.13 source 198.18.0.2
traceroute to 198.18.0.13 (198.18.0.13) from 198.18.0.2, 30 hops max, 40
byte packets
1 198.18.122.11 (198.18.122.11) 0.822 ms 0.507 ms 0.503 ms
2 198.18.113.13 (198.18.113.13) 1.244 ms 0.823 ms 0.893 ms
SW2(config)#
The next question asks you to ignore the MTU size. This is a command to overcome MTU problems
when building up an OSPF adjacency. The OSPF adjacencies are only established when certain link
properties and configuration settings are matched on both sides.
One of them is MTU size. This option however is configurable.
SW1-1
SW1-1(config-if)# ip ospf mtu-ignore
101
SW1-3
SW1-3(config-if)# ip ospf mtu-ignore
The next 2 questions in the task can be solved in different ways. You will need both different ways to
solve this to get the solution correct. This is because of the behavior OSPF has by default on Ethernet
interfaces. OSPF knows various ways of determining a network connection.
Within Classical IOS the choice of different network types was huge. Mainly due to all different serial
interfaces that were available that required a different behavior (sometimes broadcast traffic wasnt
supported for example).
Within NX-OS the choice is narrowed down to 2:
Type
Description
Broadcast
Allows for a shared network (multi-access) with multiple routers that can
become neighbors. Uses a Designated Router for centralizing
communication.
Point-to-Point
Simple point-to-point link where broadcast traffic is supported. There is no

option for more than 2 routers on one link, making it a lot faster to
converge and has no overhead in Designated Router elections.
By default all Ethernet interfaces (meaning all interfaces on an NX-OS device) are running the Broadcast
network type as it would be possible that multiple routers are sharing the same Layer 3 network
segment. This adds overhead to the set-up of the adjacency as OSPF Broadcast networks are using a
principle of a Designated Router per network segment. This Designated Router (DR) maintains
an adjacency with all routers. Other non-DR routers only have this single connection the DR router as
there OSPF adjacency. This ensures a good scaling in OSPF broadcast networks. To ensure the network is
secured to a DR failure, there is also an election of the Backup Designated Router (BDR).
102
All Full OSPF adjacencies are shown in the drawing below:
Routers use a different Multicast group to communicate to the DR and BDR routers. For normal
communication the 224.0.0.5 group is used, but for DR and BDR communications the 224.0.0.6
group is used. As shown in the drawing. Routers only communicate with the DR and BDR routers. The
Hello messages from other routers on the segment are seen, but a full adjacency will not be formed. The
routers will remain in a 2WAY type of relationship with each other.
The DR election is a process which is using a Priority value to determine which router is elected. As a tiebreaker the Router ID is used. The router with highest priority value or in case of equal priorities the
highest Router ID is elected as the Designated Router. After the DR is chosen the election is run a second
time to determine the Backup Designated Router.
The default priority value is 1.
To ensure that a router will never become a designated router you can change the priority value to 0.
When the priority is 0 the router will never participate in the DR or BDR election.
So we change the priority to 0 on the interfaces. Keep in mind that this change will flap your OSPF
adjacency.
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
interface Vlan122
ip ospf priority 0
interface Vlan221
ip ospf priority 0
ip ospf priority 0
103
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
interface Vlan123
ip ospf priority 0
interface Vlan132
ip ospf priority 0
ip ospf priority 0
Now the 2 uplink interfaces towards the SW1-1 and SW1-2 switches will be switching to the fact that
when you issue a show ip ospf neighbor. The neighbors will be shown as DR, meaning the local
switch is a DROTHER as the priority value ensures that SW2 and SW3 will never come a DR or BDR on
any segment.
Now the problem resides in the connection between SW2 and SW3. Both sides are now configured as
priority 0, meaning that neither of the switches will become a DR or a BDR. This now results in a nonfunctional network between those 2 switches. Where we still want that adjacency to be up!
SW2(config-if)# sh ip ospf nei
Neighbor ID
Pri State
Up Time Address
198.18.0.11
1 FULL/DR
00:08:14 198.18.122.2
198.18.0.3
1 TWOWAY/DROTHER 00:08:40 198.18.100.2
Interface
Vlan211
Eth1/5
We need to solve this to get the OSPF adjacency going between SW2 and SW3.
As just mentioned, OSPF knows 2 network types in NX-OS. The Point-to-Point network type is for
Ethernet links that are directly connected between each other where no other router will ever come on
the subnet. This type of network also doesnt know the process of electing a DR or BDR as the 2 nodes
are the only routers on the subnet.
When we configure this for the link between SW2 and SW3 we ensure that the routers establish a Full
OSPF adjacency and that none of them is a DR or BDR on that segment!
SW2
SW2(config-if)# interface Ethernet1/5
SW2(config-if)# ip ospf network point-to-point
SW3
SW3(config-if)# ip ospf network point-to-point
SW2(config-if)# sh ip ospf nei
104
Neighbor ID
198.18.0.11
198.18.0.3
Pri State
1 FULL/DR
1 FULL/ -
Up Time Address
00:08:14 198.18.122.2
00:08:40 198.18.100.2
Interface
Vlan211
Eth1/5
After configuring the point-to-point network type the adjacency is up again and everything is working
fine besides that we comply with all rules stated in the question of the task.
Next up is authentication. OSPF does support both MD5 and simple-text passwords for authenticating
the protocol. In real-life networks there is no good reason to use simple-text passwords, but for lab
purposes you should know how to use both.
Besides that OSPF also knows a difference in authenticating on a link basis or on an area basis. On area
basis the OSPF protocol requires the entire area to authenticate its adjacencies. This is a good way to
enforce authentication within an area. Link adjacencies are only dependent on the directly connected
routers so no other routers in the area know that there is authentication happening.
The first assignment indicates to secure all area 0 adjacencies. One important note here is that most
people are configuring authentication on all links between area 0 routers perfectly, but do forget a
important area 0 connection. This connection is the virtual-link crossing through area 2 to
connect area 264 to the network. This is also an area 0 adjacency and should therefore also be secured
with a hashed password! You will lose all points for this task when you forget to secure the virtual-link!
The authentication is configured using a number of commands under the interface.
SW1-1
SW1-1(config-if)#
SW1-1(config-if)#
authentication
SW1-1(config-if)#
<CR>
key-chain
message-digest
null
interface Vlan112
ip ospf auth
authentication-key
ip ospf authentication ?
Authentication password key-chain
Use message-digest authentication
Use null(disable) authentication
SW1-1(config-if)# ip ospf authentication message-digest

SW1-1(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure
SW1-1(config-if)#
When OSPF authentication is enabled, you will enable simple-password (clear-text) password
authentication. This is not what the question stated. We should enable a hashed version of the key as
the authentication method. This is done using the message-digest authentication method. You are
also able to configure a key chain, just like with EIGRP, but its not mandatory and in our example we
have no reason to do so. Therefore we configure the MD5 key just on the interface. Do not configure
the OSPF authentication-key, as that would configure the simple password authentication.
Probably the authentication still works, but this is because an empty md5 hash is still an md5 hash and is
a valid key for authentication. Still you would lose all the points on this question, because the
105
authentication is not done using the key as stated in the question. You will need to use the messagedigest-key configuration command.
SW1-1(config-if)# ip ospf authentication message-digest
SW1-1(config-if)# ip ospf message-digest-key 1 md5 IPexpertSecure
SW1-2
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
interface Vlan112
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 IPexpertSecure
interface Vlan211
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 IPexpertSecure
The final step is to enable authentication on the virtual-link as this is just an area 0 adjacency as all
other.
SW1-1
SW1-1(config-router-vlink)# ?
authentication
Authentication on the vlink
authentication-key
Configure the authentication key for the virtuallink
dead-interval
Dead interval
hello-interval
Hello interval
message-digest-key
Message digest authentication password (key)
no
retransmit-interval Packet retransmission interval
transmit-delay
Packet transmission delay
end
Go to exec mode
exit
pop
push
where
SW1-1(config-router-vlink)# authentication ?
<CR>
key-chain
Authentication password key-chain
message-digest Use message-digest authentication
null
Use null(disable) authentication
SW1-1(config-router-vlink)# authentication message-digest
SW1-1(config-router-vlink)# message-digest-key 1 md5 IPexpertSecure
We go into the sub-configuration of the virtual-link and configure the same parameters as you would on
an interface.
SW1-3
106
SW1-3(config-router-vlink)# authentication message-digest

SW1-3(config-router-vlink)# message-digest-key 1 md5 IPexpertSecure
The next authentication is area based, which means that adjacencies are enforced on to run
authentication. This time its using simple-text passwords, so regular authentication parameters. One
difference in this configuration is that the authentication is not enabled on the interface, only the key is
specified on the interface configuration and the authentication is enabled within the routing process.
SW1-1
SW1-1(config-router)# area 1 authentication
SW1-1(config-router)# interface Vlan122
SW1-1(config-if)# ip ospf authentication-key IPexpert
SW1-2
SW1-2(config-router)# area 1 authentication
SW2
SW2(config-if)# router ospf IPEXPERT
SW2(config-router)# area 1 authentication
SW2(config-router)#
SW2(config-router)# interface Vlan122
SW2(config-if)# ip ospf authentication-key IPexpert
SW2(config-if)# interface Vlan221
SW3
SW2(config-router)# area 1 authentication
SW2(config-router)#
SW2(config-router)# interface Vlan123
107
The next question is asking about summarization. Now summarization is different in OSPF compared to
EIGRP. Distance vector protocols send their routing table towards other routers in the network. A linkstate protocol will be default only advertise its link information. Every router needs to have a full view
on the network before it can start calculating the shortest path to all destinations in the network.
Now as earlier discussed the OSPF routing protocol uses a distance vector style way of advertising
networks to other areas. This means that we can summarize at the borders of areas, before injecting
routes into area 0.
This is done using the area-range command in the OSPF protocol configuration. We first need to
generate additional Loopback interfaces on SW2.
SW2
SW2(config)# int lo1
When we configure the routes:
198.18.128.0/24
198.18.129.0/24
198.18.130.0/24
198.18.131.0/24
We see that the third octet is changed. In binary the third octet looks like this:
10000000
10000001
10000010
10000011
108
Anything between the last 2 bits is changed in these static routes. So when we create a static route with
a mask that is 2 bits shorter than the /24 meaning a /22 we are good to go!
We need to get the networks into OSPF. This can be done in 2 ways. One by configuring the Loopback
interfaces to appear in the OSPF database and the other is by doing redistribution
Be aware that the summary configuration for both ways is very different! As for the first way the arearange command is used and for the second the summary-address command. In the question its
stated that the single entry should be seen as a Type 3 LSA in the backbone. When an External route is
summarized the summary is also advertised as a Type 5. Its therefore necessary to use the area-range
style configuration as that will inject a regular Type 3 LSA instead of a Type 5.
SW2
SW2(config)# int lo1
SW2(config-if)# ip ospf
network IPEXPERT area 1

Now SW2 is not the border router for the network. Summarization can only be done on the area border
routers. Therefore the area-range command is not effective when configured on SW2, but only on SW11 and on SW1-2. Dont forget you have 2 ABRs in area 1!
SW1-1
SW1-1(config-router)# area 1 range 198.18.128.0/22
SW1-2
SW1-2(config-router)# area 1 range 198.18.128.0/22
After enabling the area range. Other areas will now only see a single Type 3 LSA for these 4 networks
instead of the specific ranges!
The next question states to ensure the whole subnet is seen in the routing table. Isnt this always done?
Well in classical IOS this was the case, but in NX-OS the behavior of Loopback interfaces has changed.
There is not a real reason why you would want to put an other than /32 mask on a Loopback interface,
but it is possible in all Cisco platforms.
SW1-3
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
SW1-3(config-if)#
interface Loopback1
ip address 198.18.13.1/24
no shutdown
109
By default all Loopback interfaces will be advertised with a /32 mask, no matter what mask has been
configured on the Loopback interface itself.
SW1-1(config-if)# sh ip route ospf
198.18.0.2/32, ubest/mbest: 1/0

198.18.0.13/32, ubest/mbest: 1/0
198.18.13.1/32, ubest/mbest: 1/0
To overrule this behavior a special command is necessary.

SW1-3
SW1-3(config-if)# interface Loopback1
SW1-3(config-if)# ip ospf advertise-subnet
After configuring this command the route is now shown with its correct /24 mask in the routing table on
other routers.
198.18.0.2/32, ubest/mbest: 1/0
198.18.0.13/32, ubest/mbest: 1/0
198.18.13.0/24, ubest/mbest: 1/0
The next task is a question regarding OSPF areas. The OSPF protocol, as previously mentioned, enables
the use of different types of areas. This is done to introduce scaling in the routing protocol. Within an
area every router needs to know about all links on all routers. When the area grows to hundreds of
routers. There comes a point where nobody can keep up anymore. Then additional areas can be created
to lower the impact on routers. With good summarization at area borders the scaling can be enormous.
Now when you even want to lower the control-plane overhead on OSPF routers its possible to create
so-called stub areas.
There are several types of stub areas. Each with a different behavior. Below is a table explaining the
different types of areas and the behaviors that are related to it.
110
Type
Description
Stub
Does not allow external routes (Type-5 LSAs) in the area. Default
route can be advertised by the ABR to ensure full reach ability.
Totally Stub
Same as a stub area, so not allowing external routes (Type 5 LSAs).

Besides that Summary routes (Type-3 LSAs) are not allowed as well.
Meaning that there is only reach ability within the area and the ABRs
advertise a default route into the area to ensure the routers in that
area can still reach the rest of the network. This default is a Type-3
LSA. Meaning 1 Type-3 LSA is allowed in a Total Stub area.
Not-so-stubby
Same as a stub area, so not allowing external routes (Type 5 LSAs).

But do allowing ASBRs in the area. The result is that an ASBR within a
not-so-stubby area (NSSA) will inject Type 7 LSAs in the NSSA. These
will be converted to a regular Type 5 on the ABRs of that area.
Totally Not-so-stubby
Same as a not-so-stubby area, so not allowing Type 5 LSAs to exist in

the area, but allowing ASBRs. Then Type-3 LSAs are also not allowed
as in a Totally Stub area. This means a default route is injected in the
area to ensure reach ability to the rest of the network.
111
The drawing below illustrates the above table. Where ASBRs can be placed in the network. Remind that
area 0 can never be altered to a stub or not-so-stub area! What is not explained in the drawing, but is
possible is that ABRs are ASBRs at the same time.
Now back to the question. The question states the following characteristics of area 1:
No Type 5 LSA
o
No Type 4 LSA
o
This means that a Stub area needs to be used
This is also included in the Stub area. As there are no Type 5 LSAs in the area, its also
not required to have a Type 4 LSA (reach ability to the ASBR) in the area
No Type 3 LSA
o
This points to a Totally Stub area, where all Type 3 LSAs are filtered from crossing the
ABRs and a Default route is advertised in to the area. This would solve the question.
Except that the Default route that is injected into a Totally Stub area is still a Type 3 LSA.
The question clearly states that this is not allowed
We can solve this behavior. Within a Totally NSSA area we do not allow Type 3,4 and 5 LSAs. So its
equal to a Totally Stub area. The difference is that we can introduce a new type of LSA in the area which
112
is a Type 7 for the External routes that are injected when the NSSA area contains an ASBR. These LSAs
are interpreted by every router in the area and therefore they install those external routes from the
ASBR in the NSSA area.
What we can do at the ABRs is to ensure that the Default route they inject into the Totally NSSA area is
not a Type 3 LSA (as it would be by default on IOS). It can be a Type 7 LSA. Be aware that you need to
configure this and this is not supported on every IOS release as well. On NX-OS however its the default!
The configuration for this is relatively easy and is easy to spot with use of the question mark.
SW1-1
SW1-1(config-router)# area 1 ?
authentication Enable authentication for the area
default-cost
Specify default-cost for default summary LSA
filter-list
Filter prefixes between OSPF areas
nssa
Configure area as NSSA
range
Configure an address range for an area
stub
Configure area as a stub
virtual-link
Define a virtual link and its parameters
SW1-1(config-router)# area 1 nssa ?
<CR>
default-information-originate Originate Type-7 default LSA into NSSA
area
no-redistribution
Do not send redistributed LSAs into NSSA
area
no-summary
Do not send summary LSAs into NSSA area
translate
Translate LSA
SW1-1(config-router)# area 1 nssa no-summary ?
<CR>
default-information-originate Originate Type-7 default LSA into NSSA
area
no-redistribution
Do not send redistributed LSAs into NSSA
area
SW1-1(config-router)# area 1 nssa no-summary default-information-originate
SW1-2
SW1-2(config-router)# area 1 nssa no-summary default-information-originate
On the other routers within the area we do not have to configure any of the default route information.
Whats important for the adjacency to come up is that the area is configured as an NSSA area. This is
advertised within the adjacency packets. Therefore we only require the following information on the
other switches.
113
SW2
SW2(config-router)# area 1 nssa
SW3
SW3(config-router)# area 1 nssa
Before applying the NSSA configuration the routing table shows more specific routes within area 1.
SW2(config-if)# sh ip route ospf
198.18.0.11/32, ubest/mbest: 1/0
198.18.0.13/32, ubest/mbest: 1/0
198.18.113.0/24, ubest/mbest: 1/0
SW2(config-if)#
After applying the configuration every route is summarized into the default route.
SW2(config-router)# sh ip route ospf
0.0.0.0/0, ubest/mbest: 1/0
SW2(config-router)#
When checking the OSPF database, there are no Type 3, 4 or 5 LSAs visible so we accomplished this
question!
SW2# sh ip ospf database
OSPF Router with ID (198.18.0.2) (Process ID IPEXPERT VRF default)
Router Link States (Area 0.0.0.1)
Link ID
198.18.0.2
198.18.0.11
ADV Router
198.18.0.2
198.18.0.11
Age
215
215
Seq#
Checksum Link Count
0x8000000a 0xcff7
2
0x80000009 0xf0a5
1
Network Link States (Area 0.0.0.1)

Link ID
198.18.122.11
ADV Router
198.18.0.11
Age
216
Seq#
Checksum
0x80000002 0xa0b1
114
Type-7 AS External Link States (Area 0.0.0.1)

Link ID
0.0.0.0
ADV Router
198.18.0.11
Age
542
Seq#
Checksum Tag
0x80000002 0x1dcf
0
The final question of this task states to ensure that all OSPF routers do not attract traffic just after startup of the devices. This should last for 2 minutes. The command we are looking for is the max-metric
command, where you can add several values of how this max metric is configured.
The result is that once the router boots up and OSPF adjacencies come online. The router will advertise
everything with a maximum metric value until the timer expires. Default this timer is 600 seconds, but is
now adapted to 120.
SW1-1
SW1-1(config-router)# max-metric router-lsa on-startup 120
New command will take effect at next reload
SW1-2
SW1-3
SW1-4
SW2
SW2(config-router)# max-metric router-lsa on-startup 120
SW3
SW3(config-router)# max-metric router-lsa on-startup 120
And task 4 is finished!
115
Task 5: Redistribution, BFD and ECMP

Redistribution is the process to leak information between different dynamic routing protocols. This is
done to ensure a full reach ability in your whole network. Chances are that you will not run into
situations where you need to redistribute between 2 IGP protocols. In real life scenarios redistribution is
more commonly done between BGP and an IGP.
Pay close attention when using redistribution in your topology. You can easily run into issues and cause
routing loops in the network. This is especially the case for protocols without distinction in route
preference / administrative difference like OSPF. The active route is the one with the lowest
Administrative Distance. The following table describes the default values which can be changed per
protocol. The change can be done for certain routes, certain upstream routers or types of routes
(internal/external/local).
Source
Administrative Distance
Connected
Static
EIGRP Summary
External BGP (EBGP)
20
EIGRP
90
OSPF
110
IS-IS
115
RIP
120
External EIGRP
170
Internal BGP (iBGP)
200
Local BGP
220
Unreachable
255
Some protocols distinct routes that are internal to their own protocol and prefixes that were learnt via
redistribution. OSPF does not do this by default, but can be configured to ensure Type 5 LSA prefixes will
have another Administrative Distance attached. In most cases this is even recommended. Its easy to get
a routing loop when there is no distinction between the administrative distances for internal and
external routes.
116
In our network we will also run into some issues and these will cause issues (no connectivity issues).
With a normal redistribution the EIGRP learned routes will be learned back by the other router back into
EIGRP. The mechanism that EIGRP uses in this case is that Externally learned routes will have a higher
administrative distance attached to the routes and therefore select their primary internal routes as the
ones where traffic will be send to.
Within the OSPF network there will be issues with routes learning 2 destinations.
We first configure the switches for redistribution on 2 positions in the network. We are required to
configure a route-map for the redistribution to ensure only routes are advertised that you really want
to.
The drawing below illustrates what is about to happen when we do not enter some filtering information.
To ensure that routes are not redistributed back their to their own protocol.
Now we configure the switches with filtering and tagging in place so that routes that are advertised to
either EIGRP or OSPF are tagged with a value equal to the Administrative Distance (for easier
recognition, but this could be any value). On the next redistribution point this value gets rejected and
others get tagged. This way we have a very easy way to filter routes and ensure that we do not cause
loops in our redistribution scenario.
117
SW1-2
SW1-2(config)# route-map E->O deny 10
SW1-2(config-route-map)# match tag 110
SW1-2(config-route-map)# route-map E->O permit 20
SW1-2(config-route-map)# set tag 90
SW1-2(config-router)# redistribute eigrp IPEXPERT route-map E->O
SW1-2(config-router #
SW1-2(config-if)# route-map O->E deny 10
SW1-2(config-route-map)# route-map O->E permit 20
SW1-2(config-router)# redistribute ospf IPEXPERT route-map O->E
SW1-4
SW1-4(config)# route-map E->O deny 10
SW1-4(config-route-map)# route-map E->O permit 20
SW1-4(config-router)# redistribute eigrp IPEXPERT route-map E->O
SW1-4(config-router #
SW1-4(config-if)# route-map O->E deny 10
SW1-4(config-route-map)# route-map O->E permit 20
SW1-4(config-router)# redistribute ospf IPEXPERT route-map O->E
118
The drawing below illustrates what is happening in the network.
Routes being redistributed between protocols and re-redistributing routes back into their original
protocol. Due to the intelligence in most protocols this will not cause any harm and traffic will be
forwarded. Still it will cause a lot of dirt in the routing protocol databases.
119
Now we have successful redistribution and have a full reach ability throughout the network without any
loops!
The next couple tasks are regarding the Bidirectional Forwarding Detection (BFD) protocol. This protocol
ensures a very fast detection that a neighbor router is down, regardless of link status.
Usually the link status is used for detecting a neighbor down event. This is not reliable in situations
where for example a Ethernet circuit or DWDM circuit, without propagation of link-loss to client ports,
the routers will not know if something happens within the circuit.
The BFD protocol is the Cisco best practice for a fast detection mechanism. The protocol itself is just a
very basic Layer 3 protocol which sends very fast Hello messages. Once a certain threshold (by default 3)
is reached a trigger is given. The BFD protocol is independent of the routing protocol and can actually be
used for any protocol when supported (for example HSRP). Multiple protocols can be attached to the
same BFD session, making it a very lightweight protocol for detecting these neighbor loss events as only
a single session is running even when multiple protocols are used across the link between the 2 routers.
The above illustration shows the BFD session being totally independent of the OSPD adjacency running
on the router. Once the BFD session is detected to be offline, the OSPF protocol is signaled that this
specific router has gone offline and therefore OSPF cuts of the adjacency right away, before waiting on
its own Holddown-timer thresholds. Multiple protocols can be signaled via this single BFD session.
One additional advantage of the BFD protocol is that its so simple, that all Hello packets are generated
and handled within the Hardware ASICs. The Nexus 7000 and Nexus 5500 interfaces are all able to
handle BFD within the hardware and therefore BFD has no impact on the CPU load of the control-plane.
Sub-second hello timers in OSPF due have impact on the CPU load!
120
As the BFD protocol is just an IP protocol, it can also be secured, just like the mainstream dynamic
routing protocols with an MD5 hash. Detection is by default using a hello timer of 300ms and a
threshold of 3 missed hello packets, before declaring a neighbor down.
In this case we only need to ensure that failure detection of SW1-2 happens very fast. We will use the
BFD protocol for this. We first need to configure the BFD session under the interface and after that we
configure the routing protocol to converge along with the BFD protocol. As the BFD protocol is also a
Layer 3 protocol, we need to enable it on the NX-OS device first.
SW1-2
SW1-2(config)# feature bfd
Please disable the ICMP redirects on all interfaces
running BFD sessions using the command below
'no ip redirects '
We need to configure no ip redirects as a best practice. This has to do that it might be possible
for an interface to return BFD hellos with a redirect (when not properly configured). This might cause
that a lot of packets are send and need to be answered with an ICMP message. All ICMP traffic is
handled by the control-plane and not in the ASIC as BFD messages are. Therefore its highly
recommended to disable this.
SW1-2(config-if)# no ip redirects
SW1-2(config-if)# bfd interval ?
<50-999> TX interval in milliseconds
SW1-2(config-if)# bfd interval 100 ?
min_rx Minimum RX interval
SW1-2(config-if)# bfd interval 100 min_rx ?
<50-999> RX interval in milliseconds
SW1-2(config-if)# bfd interval 100 min_rx 100 ?
multiplier Configure detect multiplier for bfd sessions
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
bfd interval 100 min_rx

interface Vlan132
no ip redirects
interface Vlan112
no ip redirects
interface Vlan211
no ip redirects
interface Vlan124
no ip redirects
100 multiplier 3
100 multiplier 3
100 multiplier 3
100 multiplier 3
100 multiplier 3
121
There are a number of things that need to be configured when configuring BFD sessions. You
immediately require configuring the timers of the session. When you look a little further to the
questions in this task its stated that the sessions should detect failures within 300ms. Therefore we are
configuring hello-intervals of 100ms and a multiplier of 3. Meaning the device can miss 3 hellos before
declaring the neighbor down. With hellos of 100ms, this occurs at a maximum of 300ms, which after the
connected protocols are signaled.
What remains is that the OSPF and EIGRP protocols need to be configured for the BFD protocol and we
need to enable the interfaces that we want to use the protocol on.
If we would be configuring BGP, we would configure these options on the Neighbor sessions. The BFD
session configuration would still be done under the interface. Remember to configure the correct
routing protocols and also the correct interface configuration!
SW1-2
SW1-2(config-router)# bfd
SW1-2(config-router)# router eigrp IPEXPERT
SW1-2(config-if)# ip ospf bfd
SW1-2(config-if)# ip eigrp IPEXPERT bfd
Now we configured only 1 router. BFD consists of that both sides need to be configured before
everything works. We now configure the other sides of the interfaces that connect to SW1-2.
SW1-1
SW1-1(config)# feature bfd
SW1-1(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW1-1(config-if)# bfd interval 100 min_rx 100 multiplier 3
122
SW3
SW3(config)# feature bfd
SW3(config-router)# bfd
SW3(config-router)# exit
SW3(config-if)# no ip redirects
SW3(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW3(config-if)# ip ospf bfd
SW3(config-if)# no ip redirects
SW3(config-if)# bfd interval 100 min_rx 100 multiplier 3
SW3(config-if)# ip ospf bfd
SW1-4
SW1-4(config-router)# router eigrp IPEXPERT
SW1-4(config)# interface Vlan124
SW1-4(config-if)# ip eigrp IPEXPERT bfd
The questions also state that the BFD session between SW1-2 and SW3 should be secured using an SHA1 hash. We can configure this under the interfaces. Dont forget that there are 2 interfaces and
therefore 2 sessions between the switches, which both need to be secured.
SW1-2
SW1-2(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
SW1-2(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
SW3
SW3(config)# interface Vlan123
SW3(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
SW3(config)# interface Vlan132
SW3(config-if)# bfd authentication keyed-sha1 keyid 1 key IPexpertSecure
Which completes task 5!
123
Task 6: Layer 3 switching features

The final task regarding layer 3 routing protocols is about miscellaneous features that are possible with
the NX-OS. These features are again easily found in the Unicast Routing Guide and Interfaces
Configuration Guide in the online documentation that is also available to you when you are doing the
actual exam.
The first task is to configure a static ARP mapping on a specific interface. In Classical IOS this
configuration was done in Global Configuration mode. Within NX-OS this is done under the interface
where the ARP entry is reachable. This means that its not necessary to include an interface anymore in
the ARP command, but you only need to configure which mac addresses are found on which interface.
SW1-1
SW1-1(config-if)# ip arp ?
A.B.C.D
IP address
gratuitous Gratuitous
timeout
ARP timeout
SW1-1(config-if)# ip
E.E.E
EE-EE-EE-EE-EE-EE
EE:EE:EE:EE:EE:EE
EEEE.EEEE.EEEE
arp
MAC
MAC
MAC
MAC
198.18.112.24 ?
address (Option
address (Option
address (Option
address (Option
1)
2)
3)
4)
SW1-1(config-if)# ip arp 198.18.112.24 abcd.1234.5678

SW1-1(config-if)# sh run int vlan 112
!Command: show running-config interface Vlan112
!Time: Mon Sep 24 01:40:00 2012
version 5.1(5)
interface Vlan112
no shutdown
ip address 198.18.112.11/24
ip arp 198.18.112.24 ABCD.1234.5678
SW1-1(config-if)#
We configured the static ARP entry directly under the interface and completed the question.
Next is a bit cryptic description regarding Gratuitous ARPs. With this mechanism the switch will
broadcast an ARP entry for its own IP address and checks if anybody replies with a different MAC
address (different than its own local interface). After which it can send a warning about a duplicate IP
address on the segment.
This mechanism is enabled by default. Though one configuration option, to actively update its cache is
disabled. This is what the question asks for.
124
SW2
SW2(config-if)# int Ethernet1/5
SW2(config-if)# ip arp gratuitous ?
request Enable/Disable sending grat. arp request when duplicate address
detected
update
Enable/Disable arp cache updates for gratuitous arp
SW2(config-if)# ip arp gratuitous
The next question is also a bit cryptic which means you have to read the Unicast Routing configuration
guide pretty well to find this. The Nexus 7000 keeps a memory reservation for outstanding ARP entries
in its Forwarding Information Base (FIB). This memory is cleared after a while. This is done to protect the
memory from overflowing with ARP entries when an attack is started (flooding of MAC addresses by an
attacker to overflow the memory of switches).
This is called ARP throttling or Glean throttling (Glean ARPs are incomplete ARP entries
waiting for a reply) and is configured using special hardware commands.
SW1-1
SW1-1(config)# hardware ?
access-list
Access Control List
ejector
Card ejector functionality
fabrics
Fabrics supported in the switch
forwarding
Forwarding information
ip
IP
ipv6 (no abbrev) IPv6
proxy
Proxy Forwarding Information
rate-limiter
Configure Rate-Limiter for packets forwarded to
supervisor
SW1-1(config)# hardware ip ?
glean
Glean
verify Enable IPv4 and some IPv6 packet validation checks in hardware
SW1-1(config)# hardware ip glean throttle ?
<CR>
maximum Maximum number of entries
syslog
Threshold for syslog for number of packets hitting the entry
timeout Timeout
SW1-1(config)# hardware ip glean throttle
SW1-1(config)# hardware ip glean throttle maximum 2750
Remind to first enable the feature and then change the maximum.
The final question is regarding RFC 1191 to be enabled on all switches. A quick search through the
documentation (Ctrl-F or Cmd-F) learns that this is related to TCP Path-MTU-Discovery! Easily
enabled on all switches with a single command.
125
SW1-1
SW1-1(config)# ip tcp path-mtu-discovery
SW1-2
SW1-3
SW1-4
SW2
SW2(config)# ip tcp path-mtu-discovery
SW3
SW3(config)# ip tcp path-mtu-discovery
Task 6 is completed! Now before moving on with the next Task its recommended to take a little break
as the next task requires loading a new initial configuration on the switches. This will take some time,
approximately 10-15 minutes, as the switches need to be reloaded to load a new VDC configuration with
other mappings and pre-configuration.
126
Task 7: FabricPath and OTV

This final task of the Chapter is regarding very cool technologies called FabricPath (FP) and Overlay
Transport Virtualization (OTV). These technologies are very specific to the Nexus family and even there
dedicated to some platforms.
FabricPath is supported on the following platforms:
Nexus 5500 series
Nexus 7000 series with F1-linecards
Requires Enhanced Layer 2 license on both platforms
OTV is supported on the following platforms:
Nexus 7000 series with M1-linecards
Requires Transport Services license
We purposely excluded newer generation linecards (F2 and M2) as these are not in the hardware
blueprint of the CCIE Data Center and therefore not in scope of this workbook.
The questions are very basic formulated in this question. This is due to the fact that the technology
underneath is quite complex and very groundbreaking to begin with. Still Cisco made a great effort in
making this one of the easiest things to configure on the box.
FabricPath and OTV are literally just a few commands to enable and get it up and running, just the way
you want. Modifications can be done to the protocols, but these are limited. Also in scope of the CCIE
Data Center, the amount of topologies that can be created with these technologies is limited.
Before you know what exactly to configure to stretch the layer 2 VLAN across multiple datacenters over
a Fabric Path and OTV network, you need to know some structure of the protocols.
FabricPath
The FabricPath protocol is a Cisco proprietary protocol for making Layer 2 routed instead of switched.
This is done by eliminating ARP from the network and implementing IS-IS as routing protocol to
advertise reach ability in the network. This way there is no broadcast traffic occurring when its not
necessary. FabricPath certainly supports all various ways of Multicast and Broadcast traffic, but its sent
in a very optimal way.
The next big step is that its not necessary for all switches to have knowledge about all MAC addresses in
the network. Therefore a mechanism is introduced called Conversational MAC learning. This
mechanism is only available on the F-linecards of the Nexus 7000 switches and on the Nexus 5500 series
switches. It means that switches only remember the MAC address information, once there is a
bidirectional traffic flow. Instead of just learning the MAC address once a packet is seen from it.
127
The IS-IS routing protocol is totally invisible for the FabricPath protocol and for the CCIE configuring it.
Only when troubleshooting is necessary, certain commands related to IS-IS are required to check if
adjacencies are up and that topology information is exchanged.
The FabricPath protocol works with an additional encapsulation on the packet (the protocol kind of
looks like MPLS). The encapsulation is a FabricPath encapsulation. Which means that its an
adjusted Ethernet frame that is added in front of the packet. FabricPath is also called a MAC-in-MAC
protocol. This header is modified for FabricPath and therefore its also NOT supported to run FabricPath
over an Ethernet circuit or to have Switches in between the path. The connection between 2 FabricPath
devices needs to be a direct (dark) fiber connection to support it.
There is an IEEE initiative called TRILL that is very much alike FabricPath in many ways. The main
difference is that TRILL does use a standard Ethernet header as the encapsulation and therefore does
support that switches or Ethernet circuits are in between the TRILL devices. This enables a lot more use
of the protocol and its not necessary to have all TRILL enabled devices in your network. For example
you could upgrade your access or distribution switches to TRILL and have your core still run as standard
Ethernet switches. They switch the TRILL traffic and do not have any awareness of whats inside the
packet and what kind of high advanced technology is happening on the Edge of the network.
All this sounds very familiar when you are used to work with MPLS networks. Where also the core is a
very dumb device that just forwards packets to the right interfaces and that all intelligence of the
network is inside the network edge. This also accounts for both FabricPath and TRILL.
Below is a drawing explaining the traffic flow in-case of a known-unicast, meaning that ARPs have
successfully passed along. In case of an unknown unicast, the ARP request of the access device or endhost is flooded throughout the network. This is done in an optimal way using so-called Broadcast Trees.
Every edge switch (called leaf switches in FabricPath) uses a dedicated tree of switches, calculated using
the SPF algorithm of IS-IS where it will forward the traffic along. By using the Broadcast tree, every
switch in the network receives the packet only once and never multiple times, as could be the case with
ARPs and normal broadcasts.
The destination host will reply to the ARP and send it back along the same distribution / broadcast tree
through the FabricPath network. Once traffic starts being sent, the network learns about this flow and
will remember the traffic flow in its tables (conversational learning). In that case the first leaf switch
knows about where the packet should go and sends it unicast to the next switch. Where its
decapsulated and another lookup is done where the destination MAC address is located on that
particular switch. The packet is then forwarded out in plain Ethernet to the end-host.
The drawing explains this traffic flow in a very easy way so the flow is known to you.
128
The drawing explains the following flow:

1. PC 1 sends a packet to PC 2 across a standard Ethernet segment (VLAN)
2. Leaf 1 receives the packet and sees that its destination (PC 2) is on Leaf 2
3. Leaf 1 encapsulates the packet with a FabricPath header, with a destination address of Leaf
2 and a source address of Leaf 1
4. The FabricPath network forwards the packet based on the FabricPath header to Leaf 2
without looking at the rest of the packet headers
5. Leaf 2 decapsulates the packet and does a lookup on the Destination of the Ethernet frame
inside of the FabricPath frame
6. Leaf 2 forwards the packet out of its local interface to PC 2
By using this method of traffic forwarding the core network has no awareness of whats happening
inside of the FabricPath frame and can therefore have a very limited FIB table. The core network only
needs to know the way to the destination leaf switches, which is advertised using a modified version of
IS-IS.
Now on to the configuration. We also configure some SVIs to test the configuration and ensure hosts on
VLAN 666 can reach the rest of the network by using FabricPath. As you can see in the network drawing,
there are no direct connections between the switches with a Classical Ethernet encapsulation. So the
FabricPath network needs to work before replies are received from other hosts on the network.
129
SW2
SW2(config)# interface vlan666
SW2(config-if)# ip address 198.18.66.1/24
SW3
SW3(config)# interface vlan666
SW1-3
SW1-3(config)# interface vlan666
SW1-3(config-if)# no shutdown
SW1-4
SW1-4(config)# interface vlan666
FabricPath only works on the F1 linecards. The encapsulation and FabricPath VLANs can only live on the
F1 linecard and can never be configured on an M1 linecard. However, the routing feature of the M1
linecard can be borrowed by the F1 linecards. Traffic entering on the FabricPath interface will be
directed over the Fabric towards the M1 linecard to ensure that layer 3 traffic is properly handled by the
switch.
On the Nexus 5500 series, a similar configuration is done by using the Layer 3 daughter module
that is also inserted like an interface in the router. Therefore the layer 2 only ASICs can forward traffic to
the layer 3 module and comes back again to be re-encapsulated into the FabricPath network.
130
SW2
SW2(config)#
SW2(config)# feature-set fabricpath
SW2(config-vlan)# mode fabricpath
SW3
SW3(config)#
SW1-3
SW1-3(config)#
SW1-3(config)# feature-set fabricpath
SW1-3(config-vlan)# mode fabricpath
SW1-4
SW1-4(config)#
SW1-1
SW1-1(config)#
SW1-2
SW1-2(config)#
With enabling the fabricpath feature-set, you enable a whole suit of protocols in a single command. For
example IS-IS is started and the FP protocol itself. Whats also started is an automatic switch ID
selection. Its possible to manually configure the switch IDs (where traffic is forwarded to in the FP
frames), but the automatic selection will pick values which are unique in the network. Therefore we will
leave it as automatic in our network.
We also configure the VLAN which is to be extended through the FabricPath network (VLAN 666) as a
FabricPath VLAN. This is to configure the VLAN that its allowed to travel the FabricPath network.
By default all VLANs remain in so-called CE (Classical Ethernet) mode, so will never cross the FabricPath
network.
131
Now whats left is the configuration of the interfaces that are pointing to the other FabricPath switches.
Ensure that you do not configure any other interfaces with this encapsulation as a lot of processes will
be configured under this interface and for example IS-IS will start attempting to make adjacencies on
these interfaces.
SW2
SW2(config)#
SW2(config)# interface Ethernet1/1
SW2(config-if)# switchport mode fabricpath
SW3
SW3(config)#
SW1-3
SW1-3(config)#
SW1-3(config)# interface Ethernet2/9
SW1-3(config-if)# switchport mode fabricpath
SW1-4
SW1-4(config)#
SW1-1
SW1-1(config)#
SW1-2
SW1-2(config)#
132
After a while our interface should come up, IS-IS adjacencies have formed and our FabricPath network is
up and running!
The next and final step of this chapter is to configure the inter-datacenter communication. This is
accomplished using another Nexus 7000 specific Cisco Proprietary protocol called Overlay
Transport Virtualization.
Overlay Transport Virtualization

The OTV protocol works, from a data plane perspective, very similar as FabricPath works. The next-hop
of the MAC address-table is changed from a destination interface to a Destination IP address instead of
a Destination MAC address as it would be in a FabricPath network.
The OTV protocol uses IP encapsulation instead of Ethernet. Different from FabricPath is that OTV is
created to run over standard IP networks. This means that the only requirement you have between data
centers to extend layer 2 between them is a Layer 3 connection with IPv4 routing. Any IP connection
between the data centers is enough to ensure that OTV can function.
While FabricPath is called a MAC-in-MAC protocol. OTV can be called a MAC-in-IP protocol.
OTV is designed to support numerous data centers that share the same Overlay network. Meaning that
one VLAN can be extended across many different data centers. This means that the OTV network is
optimized to work in a Multicast enabled environment. The configuration becomes quite complex
when unicast is used. In our case with only 2 devices and a directly connected link we will certainly use
Multicast for our solution.
The unicast model works with a central node (or 2) that will be in charge of maintaining adjacencies with
all sites for spreading all control-plane information throughout the Overlay network. The model looks a
bit like the OSPF DR and BDR principle or the BGP route reflector principle, where devices
only maintain 2 connections to these devices and with nobody else. These primary devices will exchange
the information about the network to all nodes.
As previously mentioned the protocol works very similar to FabricPath in traffic flow in the data plane,
so we will not describe this in further detail. Only the final traffic flow will be shown in a diagram.
OTV requires a little more configuration than FabricPath, as it has no knowledge of the rest of the
network and needs some IP addressing to be configured. Besides that it requires an Overlay network to
be configured.
OTV also uses the IS-IS routing protocol to exchange the MAC address information between all nodes.
This information is exchanged through Multicast messages. This IS-IS protocol is a different instance
than the IS-IS protocol used for FabricPath. This is done to ensure they will never interfere with each
other.
133
The following parameters need to be configured for a successful OTV implementation:

1. IP addressing on the IP interfaces that are connected to the WAN network
2. Multicast needs to be enabled in the network and on the interface (IGMP)
3. A Multicast group needs to be selected for transferring control-plane traffic (controlgroup)
4. A block of SSM (Source Specific Multicast) groups (8) needs to be selected for transferring
multicast and broadcast traffic of the network across the Overlay.
5. VLANs that need to be extended should be known
6. A local VLAN (site VLAN) needs to be selected for local control-plane traffic within the
datacenter. Mainly used for split-brain detection and loop prevention. The local VLAN may never
be extended to other sites!
7. A site-identifier to ensure the site is uniquely numbered. When using multiple OTV
nodes at a single site, this number should match on all devices on that site.
OTV only works on the M1 linecards of the Nexus 7000 series switches. This line card is typically used in
core switch scenarios where also the layer 2 to layer 3 border is placed. Meaning that SVIs are
configured on the core switches to terminate layer 3 routing of the datacenter network.
There is one important limitation with OTV! The switch that is configured for OTV may not contain any
SVIs of the VLANs that are extended to other sites! This is a hardware limitation in the M1 line cards. In
future software and/or hardware releases of NX-OS and/or linecards its expected that this limitation is
removed. The current workaround is to use a different VDC on the Nexus 7000 just for OTV extension. In
our network we do not have an SVI of the extended VLAN configured on that switch (VDC) therefore we
do not run into this limitation.
Time for some configuration!
First we configure the required configuration that will support the OTV protocol.
SW1-1
SW1-1(config)#
SW1-1(config-if)# no switchport
SW1-1(config-if)# ip igmp version 3
SW1-1(config-if)#
SW1-1(config-if)# vlan 666
134
SW1-2
SW1-2(config)#
SW1-2(config-if)# no switchport
SW1-2(config-if)# ip igmp version 3
SW1-1(config-if)#
We are told to use the 198.18.10.0/24 subnet for a Layer 3 connection. This will be the only layer 3
connection in our topology, so we can use the entire /24 network if we want. We also enable IGMP to
ensure Multicast processing is enabled on the interface. OTV requires version 3 of IGMP to work
correctly. At this stage we also enable the VLAN that is about to be extended to the other site.
Meanwhile its also the FabricPath VLAN as the traffic coming in for this VLAN is coming in via a
FabricPath enabled network in our topology.
Next up is the OTV configuration.
SW1-1
SW1-1(config)#
SW1-1(config)# feature otv
As with all features. It needs to be enabled first.

SW1-1(config)# otv site-identifier 1
SW1-1(config)# otv site-vlan 10
We are required to configure a site-identifier for OTV. In FabricPath the switch IDs are
automatically selected, but for OTV the site-ID needs to be equal on all nodes within the same data
center (site), therefore this needs to be configured manually.
The site-vlan is used for sending local control-plane traffic between the OTV enabled switches. This
VLAN may never be extended across the Overlay to other nodes in the OTV enabled network. The
information sent in this VLAN is used for loop prevention and split-brain detection. By default VLAN 1 is
used for this. Its highly recommended to change this to a dedicated VLAN. There were no indications in
the question which value is required to be used, so we can pick any number. We also dont have any
other OTV enabled switches in this network, so the VLAN is unused in this topology, however required in
the configuration.
SW1-1(config)# interface Overlay1
SW1-1(config-if)# otv control-group 239.1.1.1
SW1-1(config-if)# otv data-group 232.11.22.0/28
SW1-1(config-if)# otv join-interface Ethernet1/10
135
The Overlay interface is the interface used for the encapsulation onto the OTV enabled network.
All OTV related configuration is done under this interface.
The control-group is the multicast group that is used for control-plane traffic sent across the OTV
network to exchange all information to other OTV nodes. This can be any multicast group that is
forwarded correctly through the multicast enabled network.
The data-group is the range of multicast groups, with a maximum of 8, on which the multicast and
broadcast traffic of the extended VLANs is sent. These groups should be from the Source Specific
Multicast (SSM) range. Again we didnt have any value given, so we can use any /28 subnet to ensure we
have 8 groups available for multicast/broadcast transmission across the OTV network.
The last option in this snippet is the join-interface. This is the IP interface (interface configured
with an IP address) that is used for sending the IP encapsulated traffic on. You can only configure 1
Ethernet interface with this command. When resiliency and/or load-balancing is required there are 2
options. The first is to use a Port-channel of M1 linecard interfaces as the join-interface. The second
is to use 2 or more Overlay interfaces with different OTV parameters, but with the same VLAN
extension configured.
SW1-1(config)# interface Overlay1
SW1-1(config-if)# otv extend-vlan 666
The final step in the configuration is to configure the VLANs that are going to be extended to the
other data center. In our topology this is VLAN 666 which is extended. All the above mentioned
parameters on the Overlay1 interface need to be configured first, before the no shutdown
command is issued. When some pieces of the configuration are missing, the system will report an error,
or the OTV network will not work properly.
Once the OTV network is discovered and converged. We have extended VLAN 666 across a FabricPath
network across 2 data centers interconnected via OTV!
136
In the final topology, the traffic flow will be as the following diagram illustrates.
The steps that are taken in the life of the packet are:
1. SW2 SVI1 (198.18.66.1) sends an ICMP ping to SW1-4 SVI4 (198.18.66.4) in a
standard/plain Ethernet/IP frame
2. The packet is encapsulated by SW2 into a FabricPath / Ethernet frame and sent to
the switch where the destination resides.
3. SW1-1 decapsulates the FabricPath header and does a lookup on the destination
address (SW1-4 SVI4) and finds an OTV IP address next-hop in its MAC address-table
4. SW1-1 encapsulates the packet in an OTV / IP frame and sends it across the OTV
network towards the destination
5. SW1-2 decapsulates the OTV header and performs a lookup on the Ethernet frame. The
destination address is a FabricPath Switch ID
6. SW1-2 encapsulates the frame in a FabricPath / Ethernet frame and sends it
towards the destination switch ID.
7. SW1-4 decapsulates the FabricPath header from the packet and performs a lookup
on the Ethernet frame. The destination interface is its local SVI
8. The local SVI receives the Plain Ethernet frame and arrived at its destination.
137
When ICMP pings between all SVI interfaces in the network are successful the task has been completed
and the chapter has been completed successfully!
The next chapter will be an in-depth overview of NX-OS High Availability features. FabricPath will come
back in this chapter to demonstrate the possibility to connect hosts in a resilient way to 2 leaf switches.
Take a break and freshen up before continuing to the next chapter!
138
Chapter 4: Data
Center Networking
High Availability
(NX-OS)
Chapter 4: Data Center Networking High Availability (NX-OS) is intended to let you be familiar with the
NX-OS High Availability features on the Nexus platforms to create a high available network. Various
types of deployments of Port-channels and Virtual Port-channels are discussed in this chapter. The
second part of this chapter focuses on First Hop Redundancy Protocols (FHRPs) and High Available
features of dynamic routing protocols. The third part focuses on a special implementation of virtual
port-channels in FabricPath networks.
We highly recommend creating your own diagram at the beginning of each lab so you are able to draw
Multiple topology drawings are available for this chapter.
139
General Rules
particular chapter
3 hours
140
Solutions
Task 1: Topology set-up
The first task of this high availability chapter is to configure the topology for this particular chapter. The
Nexus 7000 is pre-configured using VDCs (Virtual Device Contexts) so 2 switches are simulated out of
1 physical switch.
The layer 3 drawing is in this case more important as the VLANs and SVIs need to be created out of
this drawing. The drawing shows a number of VLANs that will be running between the routers. Keep in
mind that the layer 2 topology can be very different from the layer 3 topology.
Especially in this chapter we will be creating several layer 2 port-channels (link bundles) that are crossing
all of the switches. Our layer 3 topology however, looks very different than this physical layer 2
topology.
Drawing 2 below illustrates which VLANs and which SVIs need to be created on all switches.
141
The following configuration is required to finish this initial task of this chapter:
SW1-1
SW1-1(config)# vlan 112, 111, 121, 222
SW1-1(config)#
SW1-1(config)# interface Loopback0
SW1-2
SW1-2(config)# vlan 123, 111, 121, 222
SW1-2(config)#
SW1-2(config)# interface Loopback0
SW2
switch(config)# switchname SW2
SW2(config)#
SW2(config)# interface vlan 112
SW2(config)# interface Loopback0
142
SW3
SW3(config)#
SW3(config)# interface Loopback0
After this initial configuration the task is finished and we continue with configuring the layer 2
infrastructure of this topology.
Task 2: Port-Channels
The second task will be about configuring port-channels in a traditional way. The technology for
bundling links has been around for al long time. The technology ensures that multiple physical
connections are bundled into a single logical connection. This bundling is done for multiple reasons:
Redundancy
o
Increased bandwidth
o
The increased bandwidth is achieved by load balancing traffic over the 2 or more
available connections in the bundle.
Scalability
o
By separating the link bundle across multiple line-cards within a switch, redundancy can
be achieved. When links are removed or added to a bundle, this does not result in loss
of traffic and is therefore a very good failover mechanism.
Another reason for this is that when adding links to an existing bundle, it does not result
in downtime. Therefore, a bandwidth capacity increase of connections is very easily
done and does not require a lot of configuration. This makes the connection between
devices very scalable.
Cost
o
When more bandwidth is necessary between 2 devices, you could also upgrade the link
from 1Gbps to 10Gbps for example. However the cost associated with an upgrade to
such a high bandwidth connection is relatively very high, compared to adding a second
1Gbps connection.
143
There is always a turning point in which adding more connections to a bundle will cost
more (in OPEX for example) then moving over to a single higher bandwidth connection.
Loop prevention
o
Link bundles are a single connection and therefore there is no reason to run spanningtree between the individual members.
The bundling of physical links into a single logical one has many names in different types of
documentation. Vendors also use different naming Some of the names are:
Port-channels
EtherChannels
Link Aggregation Groups (LAGs)
Trunks (very difficult when people mean different things with the same word)
Aggregated Ethernets (AE)
VIFs
(Link) Bundles
etcetera
Cisco also uses different naming for their link bundles. Within NX-OS everything has been standardized
and is now referred to as Port-Channels. In our workbook we will either be talking about Port-Channels
or Link Bundles.
Link bundling is always done between 2 devices. The logical interface is still a connection between
device A and device B. The drawing below illustrates the bundling of links between 2 devices:
144
There are a number of scenarios described in this drawing.
Port-channel numbers do not need to match

o
Its not necessary that port-channel numbers match on both sides of the connections. In
no protocol these numbers are transported. When you are bundling links between
different vendors switches the numbering is usually different and sometimes not even
possible to match.
Some hardware platforms might have limitations to the numbering of the link bundles
(older Catalyst switches only supported up to 16 port-channels).
Interfaces in a bundle can be on different line cards in the switch

o
Interfaces can be mixed and matched across different line cards. Switches have built-in
mechanisms to ensure that packets arrive in the same order (in case of distributed
forwarding) as they arrive on the switch.
145
Dependent on the port-channel configuration, port-channel 3 on Switch A will be a working connection

or not
There are a number of compatibility requirements for interfaces to come online within a link bundle.
The following list describes the capabilities an interface that need to be matched on all participating
interfaces before they can be added to any bundle:
Link speed
Duplex setting
No combined layer 2 / layer 3
Layer 3 mode
o
No layer 3 sub-interfaces may exist on link bundle members
Layer 2 mode
o
Access or trunk mode
Allowed VLAN list
MTU size
Flow-Control settings
StormControl settings
Equal type/generation line card
Equal oversubscription configuration
No Private VLAN configuration
The switch will notify you when certain parameters on an interface are incorrect for the switch to add it
to a bundle. The best practice to do is to use a (previously hidden) command in NX-OS to erase all
interface configurations and reset it to the defaults:
SW1-1# sh run int e1/25
!Time: Sun Sep 30 08:12:00 2012
version 5.2(4)
switchport
switchport monitor
146
storm-control broadcast level 50.00

storm-control multicast level 50.00
storm-control unicast level 50.00
service-policy type qos input COS-LIMIT
service-policy type queuing output SHAPE_MY_QUEUE
no shutdown
SW1-1# conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1-1(config)# default interface Ethernet1/25
SW1-1(config)# sh run int e1/25
!Time: Sun Sep 30 08:12:35 2012
version 5.2(4)
SW1-1(config)#
When using this command you are sure that the interface will be added to the bundle without any
problems.
Now on to some configuration; the first 2 questions state that a link bundle should be configured
between some switches. Its best to read ahead on some of the next questions as question 3 and 4 state
some important information to bring up the link bundle.
Port-Channels are given a number to work on. The logical interface which is created uses this number to
be identified throughout the configuration. Numbering is very different per platform. Both platforms
used in our topology (Nexus 7000 and Nexus 5000) support up to 4094 port-channel numbers on the
switch. Its not said that the switches also supports that amount of port-channels to become
operational, but its easier to have your own numbering scheme than to be limited in numbering by the
switch capabilities.
SW1-1(config)# interface port-channel ?
<1-4096> Port Channel number
When no number is given, just use your own best practice for it. Ensure that you do not overlap on the
same switch and I would use the same number on both sides of the link so you never make mistakes
when configuring certain port-channels on different devices.
We will first configure the link bundle between SW1-2 and SW3 as we require additional configuration
for the first question.
SW1-2
SW1-2(config)# interface e3/5-6
147
SW1-2(config-if-range)# channel-group 2 mode on

SW1-2(config-if-range)# no shutdown
SW1-2(config-if-range)# interface port-channel2
SW1-2(config-if)# sw trunk allowed vlan 123
SW1-2(config-if)#
!Time: Sun Sep 30 08:31:54 2012
version 5.2(4)
switchport
channel-group 2
no shutdown
!Time: Sun Sep 30 08:31:56 2012
version 5.2(4)
switchport
channel-group 2
no shutdown
SW1-2(config-if)# do sh run int port-channel2
!Command: show running-config interface port-channel2
!Time: Sun Sep 30 08:32:00 2012
version 5.2(4)
interface port-channel2
switchport
SW1-2(config-if)#
As there is no information about any port-channeling protocol, we will use no protocol and just use the
static port-channeling configuration. Now one side of the connection is configured.
148
The interfaces do not use any port-channeling protocol as mentioned, but are statically configured
inside of a port-channel. This means that when both links are up. The interfaces are forced into a portchannel.
Keep in mind that this might cause strange behaviors on the other side of the connection as it will not
expect traffic to be load balancing between 2 individual links. This means that you might see errors for
MAC address flapping or even get ports that are becoming error disabled (err-disabled) because of a
potential loop in the network.
SW3
SW3(config)# default interface Ethernet1/1
SW3(config-if-range)# channel-group 2 mode on
SW3(config-if-range)# no shutdown
SW3(config-if-range)# interface port-channel2
SW3(config-if)# switchport
SW3(config-if)# sw mode trunk
SW3(config-if)# sw trunk allowed vlan 123
Now both sides of the connection are configured and therefore the port-channel has successfully come
online and the SVI interfaces can ping each other on the switches.
SW1-2(config-if-range)# sh port-channel summ
Flags: D - Down
P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended
r - Module-removed
S - Switched
R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
------------------------------------------------------------------------------Group PortType
Protocol Member Ports
Channel
------------------------------------------------------------------------------1
Po2(SU)
Eth
NONE
Eth3/5(P)
Eth3/6(P)
SW1-2(config-if-range)# ping 198.18.123.3
PING 198.18.123.3 (198.18.123.3): 56 data bytes
36 bytes from 198.18.123.12: Destination Host Unreachable
Request 0 timed out
149
After convergence the logical interface (port-channel) 2 is up and running!

Next up is the first connection. We want to configure port-channel 1 with some special parameters.
These include that the bundle should be negotiated between the switches before it comes online. This is
related to port-channel protocols like PAgP and LACP. The first protocol is an older Cisco proprietary
protocol which is no longer supported on NX-OS platforms. The standards based solution is the Link
Aggregation Control Protocol (LACP). This protocol uses a hello mechanism to identify
certain interfaces and switches between each other. When every capability is correctly advertised
between the switches the port-channel interfaces will come online. The most important thing a switch
negotiates with LACP is the system ID. This ID needs to be advertised on both of the interfaces. This
ensures that the switch is connected to one device only and that there is no loop in the network.
The hello mechanism also helps in redundancy switchovers. With a static port-channel it is required for
the interface to go down. Otherwise the link will remain in the bundle. With LACP an intelligent
mechanism ensures that when a certain amount of hello packets is missed, according to a threshold, the
link is removed from the bundle. This makes it possible that when there is equipment between the 2
switches which will not propagate link down events, the switch will still detect this by the missing LACP
packets.
In the following drawing a few scenarios are described that would cause errors in a configuration with
static port-channels, but is automatically solved when using the LACP protocol.
150
Links configured in a port-channel on only 1 side will become Individual or Down

o
Hot-standby interfaces
o
Dependent on the configuration of the OS, links that are configured in a LACP portchannel on only 1 side of the connection. They can still become online in an
Individual mode (no bundling) without any port-channeling capabilities. This is
different on NX-OS where by default links are disabled.
When interfaces are inserted into the link bundle that go over the maximum amount. A
static port-channel would reject the configuration, but in an LACP configuration it can
still be accepted (Still dependent on other maximums). When one of the active
interfaces than fails the now Hot-Standby interface will become inserted into the
bundle, so that in case of a loss of link, there is no capacity decrease in the link bundle.
Protection against wrong port-channel configuration

o
When one interface is placed in one bundle, where the other device is configured for
another configuration of bundles (like the bottom links in the drawing). The LACP port-
151
channel will go in a Suspended state, where as a static port-channel will have serious
problems!
Now on to the configuration of our LACP port-channel. The fourth question states that SW2 should
never start the negotiation of the port-channel. This refers to the 2 modes of LACP port-channeling that
can be configured. With all above information we can summarize in the 3 different modes that portchannels can be configured in.
Mode
Description
On
The default configuration. This mode does not initiate LACP messages on the portchannel. This mode is also referred to as static port-channels. Interfaces that
come online are immediately placed
Active
This mode enables the LACP protocol on this port-channel. After the interface
comes online, it immediately starts sending LACP packets across the link.
Passive
This mode also enables the LACP protocol on a port-channel. The difference with
active mode is that in this passive mode the switch waits for another LACP packet to
be received on the interface, before it replies back with an LACP message
This results in the following table of modes. Only the yellow marked fields will result in an operational
port-channel. The other mode combinations will result in a Suspended or Non functional port-channel.
On
Active
Passive
On
Yes
No
No
Active
No
Yes
Yes
Passive
No
Yes
No
Now on to the configuration of our second port-channel. SW1-1 and SW2 will be configuring a portchannel with ID 1 between each other.
SW1-1
SW1-1(config)# feature lacp
SW1-1(config)# int e3/1-2
SW1-1(config-if-range)# channel-group 1 mode active
SW1-1(config-if-range)# no shut
SW1-1(config-if-range)# int po1
SW1-1(config-if)# sw
152

SW2
SW2(config)# feature lacp
SW2(config)# default interface e1/1
SW2(config-if-range)# channel-group 1 mode passive
SW2(config-if-range)# int po1
SW2(config-if)# sw
SW2(config-if)# sw trunk allowed vlan 112
SW2 is stated to never negotiate the interface bundle. Therefore we will use the Passive mode
of the LACP port-channeling. After the configuration the port-channel comes online and the SVI
is working correctly.
SW1-1(config-if)# sh port-channel summary
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------1
Po1(SU)
Eth
LACP
Eth3/1(P)
Eth3/2(P)
N7K11-SW1-1(config-if)# do ping 198.18.112.2
PING 198.18.112.2 (198.18.112.2): 56 data bytes
Request 0 timed out
64 bytes from 198.18.112.2: icmp_seq=1 ttl=254 time=1.504
ms
ms
ms
ms
SW1-1(config-if)#
The next two questions in this task are regarding minimum and maximum interfaces in a port-channel.
The second question is regarding the hot-standby feature.
153
For various reasons its possible that you want to steer the amount of interfaces that are up in the portchannel. One of the most important reasons is that you have a network topology where the portchannels are not used for redundancy.
In the case which is illustrated in the drawing above, you would not want the port-channel to remain up
in this case. Imagine that the left-side path is chosen in this topology and one of the interfaces fails while
there is 16Gbps of traffic crossing the links. Now there would only be 10Gbps available on that
connection. The link cost/metric is not changed when a link fails and therefore dynamic routing
protocols are not aware of any changes in the network.
The 16Gbps of traffic will still be sent across the interface that now only has 10Gbps of bandwidth. In
this case you will want the network to failover to the right-side path of the drawing so the traffic can still
be sent to where it needs to go, across a 20Gbps connection.
SW1-1
SW1-1(config)# interface Port-channel1
SW1-1(config-if)# lacp min-links 2
SW2
SW2(config)# interface Port-channel1
SW2(config-if)# lacp min-links 2
Now when this feature is configured and we disable one of the member links in the port-channel. We
see that immediately the interface is disabled and no longer working.
SW1-1
SW1-1(config-if)# shutdown
154
SW2
SW2(config)# 2012 Sep 30 10:37:41 SW2 %ETH_PORT_CHANNEL-5-PORT_DOWN: portchannel1: Ethernet1/1 is down
2012 Sep 30 10:37:41 SW2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1:
first operational port changed from Ethernet1/1 to Ethernet1/2
2012 Sep 30 10:37:41 SW2 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface
Ethernet1/1 is down (Link failure)
2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN:
Interface port-channel1 is down (No operational members)
2012 Sep 30 10:37:43 SW2 %STP-6-PORT_RANGE_DELETED: Interface portchannel1 removed from vlan=112
2012 Sep 30 10:37:43 SW2 %STP-6-PORT_RANGE_STATE: new_state=disabled
interface=port-channel1 vlan=112
2012 Sep 30 10:37:43 SW2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1:
Ethernet1/2 is down
2012 Sep 30 10:37:43 SW2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel1:
first operational port changed from Ethernet1/2 to none
2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface
Ethernet1/2 is down (Initializing)
2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN:
Interface port-channel1 is down (No operational members)
SW2(config)# 2012 Sep 30 10:37:43 SW2 %ETHPORT-5-SPEED: Interface portchannel1, operational speed changed to 10 Gbps
2012 Sep 30 10:37:43 SW2 %ETHPORT-5-IF_DUPLEX: Interface port-channel1,
operational duplex mode changed to Full
2012 Sep 30 10:37:43 SW2 %INTERFACE_VLAN-5-UPDOWN: Line Protocol on
Interface vlan 112, changed state to down
2012 Sep 30 10:37:46 SW2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet2/2:
Ethernet2/2 is suspended
SW2(config)# sh port-channel summary

Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------1
Po1(SM)
Eth
LACP
Eth2/1(D)
Eth2/2(s)
SW2(config)# sh int po1
port-channel1 is down (suspended(min-links))
Hardware: Port-Channel, address: 001b.54c1.3429 (bia 001b.54c1.3429)
Encapsulation ARPA
Port mode is trunk
auto-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
155
Switchport monitor is off

EtherType is 0x8100
Members in this channel: Eth2/1, Eth2/2
Last clearing of "show interface" counters never
3 interface resets
30 seconds input rate 88 bits/sec, 0 packets/sec
30 seconds output rate 128 bits/sec, 0 packets/sec
RX
281 input packets 43984 bytes
0 jumbo packets 0 storm suppression packets
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun
0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
825 output packets 82131 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
SW2(config)#
The interface went down with a clear indication that this is because the minimum amount of links that is
configured (Default is 1) is not met in this case.
The next question is regarding the maximum amount of interfaces that may be placed into a bundle.
This command is not available in a static port-channel. The maximum amount of interfaces that can be
enabled on a Nexus 7000 or Nexus 5000 is 8 active interfaces.
When you go over this amount and try to activate a 9th link in a static bundle. It will reject the command
and will not place the link into the bundle. When this same task is done on an LACP bundle, the
command is accepted and the link is placed into a hot-standby state. This means that when one of
the active links fail in the bundle, the hot-standby link will take its place and will become active in the
link bundle.
The maximum amount of active interfaces in a link bundle remains 8, also with LACP port-channels.
Then the maximum amount of hot-standby interfaces is also 8. This means that the maximum
amount of interfaces that can be configured within a LACP port-channel is 16.
Next up we configure the maximum amount of interfaces in Port-Channel1. As we need to place
interfaces into hot-standby mode, when another interface is added. We set the maximum to the current
amount of active interfaces.
156
SW1-1
SW1-1(config)# interface Port-channel1
SW1-1(config-if)# lacp max-bundle 2
SW2
SW2(config)# interface Port-channel1
SW2(config-if)# lacp max-bundle 2
Now we add another interface into the link bundle.

SW2
SW2(config-if)# channel-group 1 mode passive
2012 Sep 30 11:02:09 N7K12-SW2 %ETH_PORT_CHANNEL-5-PORT_HOT_STANDBY: portchannel1: Ethernet1/3 goes to hot-standby
SW2(config-if)# sh port-chan summ
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------1
Po1(SU)
Eth
LACP
Eth1/1(P)
Eth1/2(P)
Eth1/3(H)
We see that this is accepted, but the interface is placed in hot-standby mode, because the maximum
amount of interfaces is set to 2.
The next question is regarding a new port-channel between SW2 and SW3. This interface is not used as a
layer 2 connection. This one should be configured as a Layer 3 routed interface. Last thing is that
the interface should be negotiated by both sides. In an earlier question it was stated that SW2 should
never be actively negotiating its connection. So SW2 should be set on passive mode.
In production networks its a best practice to always configure both sides to be active.
SW2
SW2(config-if-range)# channel-group 3 mode passive
SW2(config-if)# no switchport
157
SW3
SW3(config-if-range)# channel-group 3 mode active
To verify the layer 3 connection we ping the other end.

SW2(config-if)# ping 198.18.23.3
PING 198.18.23.3 (198.18.23.3): 56 data bytes
Request 0 timed out
N7K12-SW2(config-if)#
Port-channel 3 is configured successfully!

The next question is referring to a behavior that has changed in NX-OS compared to Classical IOS. In
Classical IOS when an interface that was brought up in a LACP port-channel without any configuration on
the other end of the link. Meaning that it would not receive any LACP packets from the other device.
The link would still become available, but in a so-called Individual state. This means the link would
become available in a normal link state, unbundled. The Port-channel interface would not see this
interface as a member until it receives LACP packets.
Within NX-OS this behavior changed. When the interface comes online on an NX-OS interface by default
the interface will be in Suspended mode and not come online. This is the better behavior as the
Individual interfaces might cause issues and can even cause loops.
SW2
SW2(config-if)# int po3
SW2(config-if)# no lacp suspend-individual
ERROR: Cannot set/reset lacp suspend-individual for port-channel3 that is
admin up
SW2(config-if)# shut
Warning: !! Disable lacp suspend-individual only on port-channel with edge
ports. Disabling this on network port port-channel could lead to loops.!
158
SW2(config-if)#
The switch already warns you that the Individual state might cause loops in the network, so the
best practice is to leave this enabled!
SW3
SW3(config-if)# int po3
Warning: !! Disable lacp suspend-individual only on port-channel with edge
ports. Disabling this on network port port-channel could lead to loops.!
SW3(config-if)#
To test the behavior we will disable the LACP on one of the links and see what happens.
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int e1/5
no channel-group 3
int e1/6
no channel-gr 3
On the other side of the link, the link went into suspended mode.
SW3(config-if)# do sh port-chan sum
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------3
Po3(RD)
Eth
LACP
Eth1/5(s)
Eth1/6(s)
When the link is configured like stated in question 9. The interfaces are up in an individual state
and the port-channel remains down.
N7K12-SW2(config-if)# sh port-chan summ
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------Copyright by IPexpert. All rights reserved.
159
Po3(RD)
Eth
LACP
Eth2/1(I)
Eth2/2(I)
SW2(config-if)# sh int e1/5 | in 1/5

Ethernet1/5 is up
SW2(config-if)# sh int e1/6 | in 1/6
Ethernet1/6 is up
SW2(config-if)# sh int po3 | in port-chan
port-channel3 is down (No operational members)
SW2(config-if)#
As there are no members of the port-channel that received LACP packets. The channel remains down
and the individual interfaces are up!
Question 10 relates to the selection of ports within a Port-Channel. This selection is crucial for which
interfaces become Participating in a bundle and which ones become hot-standby. It has also
to do which interface is selected as the Primary interface in a bundle, where the major amounts of
LACP packets are sent through.
As the maximum amount of links in a bundle is 8, the maximum capacity of the bundle is 80Gbps.
This question states that one of the links should always be chosen to participate when its active and
one should be selected as hot-standby immediately after new interfaces are added to the bundle.
The mechanism that is used for this task is LACP port-priority. This is a similar priority value to
the Spanning-Tree port-priority value where a port on a local switch is selected. This value is never
transported outside of the switch.
The lower the value that is configured. The higher priority is assigned to an interface. Therefore we give
Ethernet1/5 a very low value (lowest possible) to ensure that it is always chosen.
SW2
SW2(config-if)# lacp port-priority 1
SW3
The default priority value is 32768. This means that if we want the interface to become hot-standby as
soon as an additional interface is added to the link-bundle this value should be increased.
SW2
SW3
160
The next question states that Port-channel3 should be using a very fast detection mechanism. Within
NX-OS its not possible to change the LACP timers by configuring specific values. It is possible however to
use sub-second hello messages and have a timeout timer of 1 second to ensure that a neighbor is
signaled as down after 1 second.
This is done under individual interfaces and the interface needs to be disabled before this is turned on!
SW2
SW2(config-if)# int e1/5
SW2(config-if)# lacp ?
port-priority Set LACP port priority
rate
Configure rate at which PDUs are sent by LACP
SW2(config-if)# lacp rate ?
fast
Specifies that LACP control packets are transmitted at the fast
rate, once every 1 second
normal Specifies that LACP control packets are transmitted at the
normal rate, every 30 seconds
after the link is bundled
SW2(config-if)# lacp rate fast
ERROR: Cannot set lacp rate. Port Ethernet1/5 is not admin down in portchannel
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int e1/5
shut
lacp rate fast
no shutdown
int e1/6
shut
lacp rate fast
no shut
The next question requires the configuration of load balancing parameters. This is a global setting for
the entire switch. It can also be configured per line card. However there are very few cases in which this
would be recommended. The global setting is relevant for outgoing traffic only. There is not option to
control the way traffic is incoming on the switch itself.
The load-balancing can be a crucial setting on the switch, but is very dependent on what is connected to
the port-channels that are configured on the switch. The configuration is dependent on the hardware
platforms. Some platforms might not support all types of load balancing. Within the NX-OS platform,
161
they are all based on high-end switching platforms. So both the Nexus 7000 and the Nexus 5000
platforms in our topology support all of the options.
The load balancing is accomplished using a hash of various kinds of information that is found in the
header of a packet. This hash is generated through a combination of multiple values found in the Layer
2, Layer 3 and/or Layer 4 header.
The load-balancing options are separated in multiple categories. For layer 2 ethernet traffic a different
load balancing is applied than for layer 3 IP traffic.
The following options are available for Layer 3 IP load-balancing:
Source VLAN number
Destination VLAN number
Source and destination VLAN number
Destination IP address
Source IP address
Source and destination IP address
Source TCP/UDP port number
Destination TCP/UDP port number
Source and destination TCP/UDP port number
The following options are available for Layer 2 Ethernet load-balancing:
Destination IP address and TCP/UDP port
Destination IP address, TCP/UDP port and VLAN
Destination IP address and VLAN
Destination MAC address
Destination TCP/UDP port
Source & Destination IP address and TCP/UDP port
Source & Destination IP address, TCP/UDP port and VLAN
Source & Destination IP address and VLAN
162
Source & Destination MAC address
Source & Destination TCP/UDP port
Source IP address and TCP/UDP port
Source IP address, TCP/UDP port and VLAN
Source IP address and VLAN
Source MAC address
Source TCP/UDP port
The NX-OS platform does not care when traffic that is sent on one interface and received on another
interface. Therefore its not required to have an equal setting on both sides of the interfaces, but it is
recommended to have it equal.
A simple mathematic calculation can be done that there when more fields are taken into the hash, the
more different hashes are generated. This means in a more granular load balancing. The load balancing
based on fields in the IP header also takes care of that all traffic that is related to a single traffic flow is
always sent on a single interface. This overcomes the issue when the approach would be a per-packet
load balancing scheme that traffic can arrive in a different order.
Having the traffic flows sent across a single interface helps in the performance of the application.
In case of layer 3 connections there is no reason to do load balancing across MAC addresses, because
the routed interfaces are the only 2 MAC addresses that are used in the IP subnet. Therefore these
options are not available in the Layer 3 load balancing options.
To complete this task its stated that the most information should be used for load balancing traffic
across the member links in a port-channel.
On the Nexus 7000 this command cannot be issued from a Virtual Device Context (VDC). Only the
default VDC can change the load balancing algorithm for the switch and modules. In our question we
only need to change the Nexus 5000 switches.
Remember that layer 2 and layer 3 load balancing needs to be changed as we have both layer 2 and
layer 3 Port-Channels on these switches.
SW2
SW2(config)# show port-channel load-balance
Port Channel Load-Balancing Configuration:
System: src-dst ip
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: src-dst mac
163
IP: src-dst ip
SW2(config)# port-channel load-balance ?
dst
Destination based parameters
ethernet Ethernet port-channel
src
Source based parameters
src-dst
Source-destination based parameters
SW2(config)# port-channel load-balance ethernet ?
dest-ip-port
Destination IP address and L4 port
dest-ip-port-vlan
Destination IP address, L4 port and
destination-ip-vlan
Destination IP address and VLAN
destination-mac
Destination MAC address
destination-port
Destination L4 port
source-dest-ip-port
Source & Destination IP address and
source-dest-ip-port-vlan Source & Destination IP address, L4
VLAN
source-dest-ip-vlan
Source & Destination IP address and
source-dest-mac
Source & Destination MAC address
source-dest-port
Source & Destination L4 port
source-ip-port
Source IP address and L4 port
source-ip-port-vlan
Source IP address, L4 port and VLAN
source-ip-vlan
Source IP address and VLAN
source-mac
Source MAC address
source-port
Source L4 port
VLAN
L4 port
port and
VLAN
SW2(config)# port-channel load-balance ethernet source-dest-ip-port-vlan

First we configure the Layer 2 Ethernet load balancing using the ethernet
statement.
SW2(config)#
SW2(config)#
SW2(config)# port-channel load-balance ?
dst
Destination based parameters
ethernet Ethernet port-channel
src
Source based parameters
src-dst
Source-destination based parameters
SW2(config)# port-channel load-balance src-dst ?
ip
IP
ip-l4port
IP and L4 port
ip-l4port-vlan IP, L4 port and VLAN
ip-vlan
IP and VLAN
l4port
L4 port
mac
MAC
SW2(config)# port-channel load-balance src-dst ip-l4port-vlan
SW2(config)#
164
The other configuration options (src, dst, src-dst) are used for the Layer 3 IP load-balancing.
SW3
SW3(config)# port-channel load-balance ethernet source-dest-ip-port-vlan
SW3(config)# port-channel load-balance src-dst ip-l4port-vlan
We verify the new configuration and now we are using the best load balancing possible.
SW2(config)# show port-channel load-balance
Port Channel Load-Balancing Configuration:
System: src-dst ip-l4port-vlan
Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: src-dst mac
IP: src-dst ip-l4port-vlan
SW2(config)#
Before we can continue with the next task we need to remove the port-channel configuration between
the switches as we will be re-using those interconnecting links for exciting new virtual Port-Channels!
SW1-1
SW1-1(config)# no interface port-channel 1
SW1-2
SW1-2(config)# no interface port-channel 2
SW2
SW2(config)# no interface port-channel 1
SW3
SW3(config)# no interface port-channel 2
165
Task 3: Virtual Port-Channels (vPCs)

The third task is about a unique feature only available on the Nexus series platforms. The building of socalled Virtual Port-Channels is only available on the Nexus 7000 and Nexus 5000 platforms.
Similar technology is however possible on Catalyst 3750 and 6500 series switches, although it has
absolutely no connection to the vPC technology.
This technology has been engineered to solve a very common problem. One of the reasons to
implement Port-Channels is to deliver redundancy to the interfaces. As stated in the previous task the
failover to a remaining interface or the insertion of a new link in the Port-Channel does not result in any
traffic loss. This ensures a very smooth operational scenario where links can be easily expanded and
failover occurs without any impact.
The redundancy however is not well engineered as the link is still made between 2 single equipped
devices. If one of the 2 (core) devices fails, the entire link goes down as illustrated by the drawing below.
The connecting device needs a manual switchover to the other link or another failure mechanism like
Spanning-Tree to use the other link.
To overcome this redundancy issue we want to use a technology to be able to create a Port-Channel
between 2 different switches that simulate to be one logical switch with one logical link between the
host/switch and itself.
There are multiple names for this technology, all resulting in the same basic principle. A few examples
are:
Cross-stack-member (StackWise) Port-Channels on the Catalyst 3750 series
Virtual Switching System (VSS) on the Catalyst 6500 series
Other vendor implementations of stacking/virtual chassis/IRF/etc.
166
One generic vendor-independent term used in this case is Multi-Chassis Link Aggregation Groups (MCLAG).
The Virtual Port-Channeling technology is the only technology that leverages a different approach than
other implementations of MC-LAG. The vPC technology is the only one that uses separated controlplanes for both switches. In other implementations the control-planes are merged so you only have a
single management entity to mange.
In case of the vPC technology you keep 2 separate switches (control-planes) that have shared some
features with each other. This has a very special reason why this is done on the Nexus series. The Nexus
series are also capable of transferring FCoE traffic, which is explained in more detail in a later chapter.
The Fibre-Channel protocol requires the separation of 2 fabrics. The traffic that uses Fabric A is never
allowed to cross any devices that are related to Fabric B. For this reason the vPC technology does not
combine the control-planes of the switches so you can keep your Fibre Channel (oE) fabrics separated!
The technology uses a number of components to be able to function correctly and to ensure a loop free
topology for the vPC connected devices.
A vPC identifies itself towards and end-host as a normal LACP or Static port-channel. The end device has
no indication that it is connected to 2 different switches and therefore assumes that its a normal PortChannel. The technology only operates on the Nexus switches. Before a vPC will come online a number
of topics need to be in place:
vPC Domain configuration

o
The vPC domain configuration configures the role of the device (primary / secondary)
and the Domain ID.
The Domain ID needs to match on both devices that are able to create vPCs between
each other
vPC Peer-Link
o
The vPC peer-link is by best-practice a Port-Channel of 2 times 10Gbps between the 2

vPC peer devices. This link is used for control information, synchronization and data
traffic when traffic needs to exit out of another vPC peer device
vPC Peer Keepalive link

o
The vPC peer keepalive link is also a very crucial part of the set-up. Without a dedicated
keep-alive link the vPCs will not come online. This link is used for simple hello packets
(can be send across a layer 2 or a layer 3 network).
The Hello packets are required for split-brain detection, in case the Peer-Lin k
fails, one of the vPC peer devices (the secondary) will shut down its vPC ports
167
You require a dedicated GigabitEthernet interface or you can use the out-of-band
management Ethernet port (mgmt0)
The drawing below illustrates the terminology of the mentioned required topics:
All topics are discussed in the questions in this task. The initial question is to ensure that SW1-1 and
SW1-2 will form a pair to create vPCs. Again its good to read ahead as we require some additional
information regarding the set-up of our peer-link and peer keepalive link.
We need all this information before the vPCs will be able to form and before we will see information
regarding the switches forming an adjacency.
The information that is exchanged between the vPC switches uses the Cisco Fabric Services over IP
protocol (CFS). This protocol has been used by the Cisco MDS (fibre channel) switches for years to
exchange information over the Fibre Channel protocol about the other switches. Using this protocol
Cisco has been able to extend the Fibre Channel features with this synchronization protocol. The IP
version of this protocol has been extended to support vPCs on the Nexus series.
There are some limitations on the vPC technology. These are a few of the most important limitations:
No mix of Nexus models

o
You cant mix Nexus 5000 and Nexus 7000 switches to be in the same vPC domain
No mix of line card types / generations

o
On the Nexus 7000 you cannot mix line card types. When the first switches uses M1 line
cards, the second switch should also use an M1 line card for both the vPC Peer Link and
the vPC client interfaces
168
There are only 2 switches in a domain

o
You cant have more than 2 switches in a single vPC domain.
A switch can only belong to 1 domain at any given time.
Only Layer 2 port-channels

o
You cant create a Layer 3 port-channel which is a vPC
Now on to some configuration for the 2 domains that we are going to configure.
We enable the vPC functionality first and need to ensure that SW1-1 is the Primary switch in the vPC
configuration and SW1-2 the secondary.
SW1-1
SW1-1(config)# feature vpc
SW1-1(config)# vpc domain 100
SW1-1(config-vpc-domain)# role ?
priority Configure priority to be used during vPC role
(primary/secondary)
election
SW1-1(config-vpc-domain)# role priority 100
Warning:
!!:: vPCs will be flapped on current primary vPC switch while attempting
role change ::!!
Note:
--------:: Change will take effect after user has re-initd the vPC peerlink ::--------
SW1-2
SW1-2(config)# feature vpc
SW1-2(config)# vpc domain 100
SW1-1(config-vpc-domain)# role priority 1000
Warning:
!!:: vPCs will be flapped on current primary vPC switch while attempting
role change ::!!
Note:
--------:: Change will take effect after user has re-initd the vPC peerlink ::--------
We use the Role Priority command to determine which switch will be the primary and which switch will
be the secondary switch in our vPC configuration. Like the switch mentions. When you are doing this in
an active configuration. The peer-link should be flapped to ensure the new roles are assigned. This has
to do with the nature of the role. Its not pre-empted, meaning that it will not flap when it sees a peer
come online with a better priority value. Only after re-initialization the roles are (re-)assigned. In this
case we make sure that SW1-1 has a much lower priority value (and therefore a higher priority) than
SW1-2.
169
On SW2 and SW3 we do not care about the role of the devices in the vPC configuration. Remember that
when you are configuring vPCs between 2 different sets of Nexus switches like in our topology. You
need to use different Domain IDs. Otherwise the vPCs will not come online when equal Domain IDs
are used between the 2 sets of switches.
SW2
SW2(config)# feature vpc
SW2(config)# vpc domain 200
SW3
Next up are the configurations for the peer keep alive links. These should always be configured before
you go to the vPC Peer-link configuration. The vPC Peer-link will not come online when no hello
messages are received over the peer keep alive link. This link is crucial in the set-up!
On the Nexus 5000 switches we will be using the mgmt0 interface. On the Nexus 7000 its the best
practice to use a dedicated GigabitEthernet interface from the line cards instead of the management
interface on the supervisor.
SW2
SW2(config-vpc-domain)# peer-keepalive destination 172.16.100.3 vrf
management
SW3(config-vpc-domain)#
SW3
SW3(config-vpc-domain)# peer-keepalive destination 172.16.100.2 vrf
management
SW3(config-vpc-domain)#
Remember to always specify the VRF that is used for these keep alive messages. As a best practice they
should never be sent from the Global Routing Table. On the Nexus 5000 we use the mgmt0 interface for
our keepalives.
After configuration, the vPC protocol has not been successfully established, but we do see another peer
and mention that it is alive!
SW2(config-vpc-domain)# sh vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id
Peer status
: 200
: peer link not configured
170
vPC keep-alive status

:
Configuration consistency status :
Per-vlan consistency status
:
Configuration inconsistency reason:
Type-2 consistency status
:
Type-2 inconsistency reason
:
vPC role
:
Number of vPCs configured
:
Peer Gateway
:
Dual-active excluded VLANs
:
Graceful Consistency Check
:
Auto-recovery status
:
peer is alive
failed
failed
vPC peer-link does not exist
failed
vPC peer-link does not exist
none established
0
Disabled
Disabled (due to peer configuration)
Disabled
The next question is about configuring the vPC peer keepalives across a dedicated SVI and the traffic
should be separated, like the best practice, from the global routing table. We do this by using a
dedicated VRF for the keepalives. The question did not specify any VLAN number, so we can use
anything we want. As all other IP subnets keep their VLAN number equal to the third octet, we will keep
this standardization to make it easier for ourselves.
We configure the SVI and VRF first and then configure the peer-keepalive statement:
SW1-2
SW1-2(config-if)# ping 198.18.5.11
PING 198.18.5.11 (198.18.5.11): 56 data bytes
Request 0 timed out
64 bytes from 198.18.5.11: icmp_seq=1 ttl=254
time=1.518
time=0.825
time=0.838
time=0.731
ms
ms
ms
ms
SW1-2(config-if)#
We can ping the other end successfully, but we also need to configure the VRF.
SW1-2(config-if)# vrf context keepalives
SW1-2(config-vrf)# int vlan 5
SW1-2(config-if)# vrf member keepalives
% Deleted all L3 config on interface Vlan5
171
Be aware that all IP addressing and other Layer 3 configuration is removed from the interface once its
placed into a VRF!
SW1-2(config-if)# ping 198.18.5.11 vrf keepalives
PING 198.18.5.11 (198.18.5.11): 56 data bytes
Request 0 timed out
ms
ms
ms
ms
SW1-2(config-if)#
SW1-2(config-if)# vpc domain 100
SW1-2(config-vpc-domain)# peer-keepalive destination 198.18.5.11 vrf
keepalives
ERROR: Management IP cannot be used as source IP for non-default VRF,
please enter source IP
keepalives source 198.18.5.12
By default the management IP address is used as the source address for the vPC keep alives. Therefore
its required to configure the source IP address when you are using another interface than the mgmt0.
SW1-1
SW1-1(config-if)# vrf context keepalives
SW1-1(config-vrf)# int vlan 5
SW1-1(config-if)# vrf member keepalives
keepalives source 198.18.5.11
SW1-1(config-vpc-domain)#
The last required configuration before we can bring up the vPCs itself are the vPC peer-links. In one
example we will be using a port-channel (standard) to ensure that a link can fail within the peer-link
before its resulting in erroneous operational issues. This is however not required and we can also
configure just a single link as the vPC peer-link.
Remember that the peer-link might also carry data traffic, so scaling the link in terms of uplink capacity
is very important.
172
First we create a single link peer-link between SW1-1 and SW1-2. The peer-link should either be a
FabricPath enabled interface or a Layer 2 trunk interface with all VLANs enabled.
SW1-1
SW1-1(config-if)# vpc peer-link
SW1-2
SW1-2(config-if)# vpc peer-link
The second example is configuring the peer-link as a port-channel. We first create a standard LACP
based port-channel between the 2 switches and enable the vPC peer-link functionality on these
interfaces.
SW2
SW2(config-if-range)# exit
SW2(config)# interface port-channel23
SW2(config-if)# vpc peer-link
SW3
SW3(config-if-range)# exit
After the configuration the vPC peer-link is signaled up and working and the switches are ready to be
configuring vPC port-channels!
SW2(config-if)# sh vpc brief
Legend:
vPC domain id
Peer status
Configuration consistency status
:
:
:
:
:
:
200
peer adjacency formed ok
peer is alive
success
success
Consistency Check Not Performed
173
vPC role
Peer Gateway
:
:
:
:
:
:
primary
0
Disabled
Enabled
Disabled
vPC Peer-link status

--------------------------------------------------------------------id
Port
Status Active vlans
---------- -------------------------------------------------1
Po23
up
1
The next question asks to ensure that the vPCs remain up and running when a peer fails or reboots. The
split-brain detection works in a way that the secondary switch turns its vPCs off when it suspects
a split-brain. To recover from this you can force the switch to bring up its VDCs after a time-out.
SW1-1
SW1-1(config-vpc-domain)# auto-recovery ?
<CR>
reload-delay Duration to wait before assuming peer dead and restoring
vpcs
SW1-1(config-vpc-domain)# auto-recovery reload-delay ?
<240-3600> Time-out for restoring vPC links (in seconds)
SW1-1(config-vpc-domain)# auto-recovery reload-delay 300
Warning:
Enables restoring of vPCs in a peer-detached state after reload, will
wait for 300 seconds to determine if peer is un-reachable
SW1-2
SW1-2(config-vpc-domain)# auto-recovery reload-delay 300
Remember that there are 2 features that do almost the same. Before NX-OS 5.2(1) the command
used to solve this was reload restore. After this release it has been deprecated and replaced with
auto-recovery.
The reload restore command did not solve the issue when a peer failed, only in case of a reboot.
Therefore in this question the only right answer is the new version of the command.
The spanning-tree question is related to vPC configuration where the vPC switches will identify
themselves as a single Spanning-Tree switch to the connecting switches that think they are connecting
with a single switch. Therefore the spanning-tree configuration that is made locally only applies to that
174
specific switch and not to connecting switches that are using vPCs. Except when the peer-switch
configuration is applied. Then the CFS protocol is used to push the spanning-tree control plane
messages to the other switch. You still need to configure the spanning-tree settings independently, but
when they match. The switch will be identified as a single root bridge towards the rest of the network.
SW2
SW2(config-if)# vpc domain 200
SW2(config-vpc-domain)# peer-switch
SW3
SW3(config-if)# vpc domain 200
SW3(config-vpc-domain)# peer-switch
Now to verify we check the spanning-tree output of VLAN 1 for both switches. We see a default priority
value is set.
SW2(config)# sh spanning-tree
VLAN0001
Spanning tree enabled protocol rstp
Root ID
Priority
32769
Address
0023.04ee.be64
This bridge is the root
Hello Time 2 sec Max Age 20 sec
Bridge ID
Priority
Address
Hello Time
Forward Delay 15 sec
32769 (priority 32768 sys-id-ext 1)

0023.04ee.be64
2 sec Max Age 20 sec Forward Delay 15 sec
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Po23
Desg FWD 1
128.4118 (vPC peer-link) Network P2p
SW3# sh spanning-tree
VLAN0001
Root ID
Priority
32769
Address
0023.04ee.be64
Bridge ID
Priority
Address
Hello Time
32769 (priority 32768 sys-id-ext 1)

0023.04ee.be64
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Po23
Root FWD 1
175
N7K-8-1#
Now we change the priority to the asked 8192 and check again.
SW2
SW2(config-if)# spanning-tree vlan 1 prio 8192
SW3
SW3(config-if)# spanning-tree vlan 1 prio 8192
Now we need to verify again.

SW2(config)# sh span
VLAN0001
Root ID
Priority
8193
Address
0023.04ee.be64
Bridge ID
Priority
Address
Hello Time
8193
(priority 8192 sys-id-ext 1)
0023.04ee.be64
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Po23
Desg FWD 1
SW3(config)# sh span
VLAN0001
Root ID
Priority
8193
Address
0023.04ee.be64
Bridge ID
Priority
Address
Hello Time
8193
(priority 8192 sys-id-ext 1)
0023.04ee.be64
Interface
Role Sts Cost
Prio.Nbr Type
---------------- ---- --- --------- -------- ------------------------------Po23
Root FWD 1
Although the switches both identify as root. SW3 also has a root port as SW2 is the true root bridge in
this network. Still SW3 will recognize itself as the root bridge and will act like it for the rest of the
network!
176
The next task is one where we finally begin configuring the vPCs!
The initial connection is made between vPC domain 100 and a client switch, which is SW2 in this case.
From SW2s perspective this is a standard port-channel towards another standard switch.
We first configure the port-channel on SW2. We use LACP in this case, but we couldve used a static
Port-Channel (mode on) as well. The best practice however is to use LACP port-channels.
SW2
SW2(config-if)# int e1/1-2
SW2(config-if-range)# int port-channel101
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------101
Po101(SD)
Eth
LACP
Eth1/1(I)
Eth1/5(I)
SW2(config-if)#
The port-channel is currently not operational because we havent configured the other side yet.
SW1-1
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
int e3/1
channel-group 101 mode active
no shut
int port-channel101
sw mode trunk
vpc 101
no shut
SW1-2
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int e3/3
no shut
int port-channel101
sw mode trunk
vpc 101
no shut
177
SW1-2(config-if)#
The only thing specific about this configuration is the vpc command. This command needs to be the
same on both vPC peer switches. Port-channel numbers do not need to match, but vpc numbers do! Its
highly recommended to keep the port-channel number and vpc number equal, because it eases your
troubleshooting a lot! There is also no reason why these should be different.
After configuring the vPC peer switches we see the port-channel (standard) is now up and running on
SW2.
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------101
Po101(SU)
Eth
LACP
Eth1/1(P)
Eth1/5(P)
On the vPC switches we also see a perfectly functioning port-channel, except with just one member.
SW1-1(config-if)# sh port-chan sum
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
Channel
------------------------------------------------------------------------------101
Po101(SU)
Eth
LACP
Eth3/1(P)
SW1-2(config-if)# sh port-chan sum
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
------------------------------------------------------------------------------Group PortType
178
Channel
------------------------------------------------------------------------------101
Po101(SU)
Eth
LACP
Eth3/3(P)
The most important verification command is to see if the vPC has been formed OK and if the state
information is correctly synced between the switches.
SW1-1(config-if)# sh vpc brief
Legend:
vPC domain id
Peer status
Configuration consistency status
vPC role
Peer Gateway
:
:
:
:
:
:
:
:
:
:
:
:
100
peer is alive
success
success
Consistency Check Not Performed
secondary
1
Disabled
Enabled
Disabled

--------------------------------------------------------------------id
Port
Status Active vlans
---------- -------------------------------------------------1
Po23
up
1
vPC status
---------------------------------------------------------------------id
Port
Status Consistency Reason
Active vlans
---------- ----------- ----------------101 Po101 up
success
success
1
The vPC is up and running! A lot of things can go wrong in your configuration before the vPC is up and
running. For example, the allowed VLAN list should be 100% equal on both sides, or the vPC will stay
down with an error message. The show vpc command will show you a lot of information regarding
the reason of the vPC being down. Usually its very easily spotted when you issue a show running for
both interfaces and see that there are differences in the configuration.
The next vPC that needs to be configured is very much the same as the previous one. Except that this
one is using vPC domain 200 instead of 100. This means that both Nexus 5000s are used as vPC peers
and the Nexus 7000 is the client switch where the vPC is connected to.
The configuration is straight forwards and a good practice for another vPC connection:
179
SW1-2
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int e3/5,e3/7
channel-group 102 mode on
no shut
int port-channel102
sw mode trunk
no shut
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int e1/3
no shut
int port-channel102
sw mode trunk
vpc 102
no shut
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int e1/3
no shut
int port-channel102
sw mode trunk
vpc 102
no shut
One important command to check is the following command, giving an overview of all settings that are
checked on the ports before the vPC comes online.
SW2(config-if)# sh vpc consistency-parameters vpc 102
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
-----------------Shut Lan
STP Port Type
STP Port Guard
STP MST Simulate PVST
mode
Speed
Duplex
Port Mode
Native Vlan
MTU
Admin port mode
Allowed VLANs
Type
----
Local Value
Peer Value
---------------------- -----------------
1
1
1
1
1
1
1
1
1
1
1
-
No
Default
None
Default
on
10 Gb/s
full
trunk
1
1500
No
Default
None
Default
on
10 Gb/s
full
trunk
1
1500
1-3967,4048-4093
1-3967,4048-4093
180
Local suspended VLANs

SW2(config-if)#
When we, for example, change the Native VLAN on the port-channel on one of the vPC peers. The vPC is
brought down, because traffic is no longer guaranteed to be treated as it should on each of the vPC
peers. This check is pro-active and happens right after the command has been issued.
SW2
SW2(config-if)# interface port-channel102
SW2(config-if)# switchport trunk native vlan 5
SW2(config-if)# sh vpc consistency-parameters vpc 102
Legend:
Type 1 : vPC will be suspended in case of mismatch
Name
-----------------Shut Lan
STP Port Type
STP Port Guard
STP MST Simulate PVST
mode
Speed
Duplex
Port Mode
Native Vlan
MTU
Admin port mode
Allowed VLANs
Local suspended VLANs
Type
----
Local Value
Peer Value
---------------------- -----------------
1
1
1
1
1
1
1
1
1
1
1
-
No
Default
None
Default
on
10 Gb/s
full
trunk
5
1500
No
Default
None
Default
on
10 Gb/s
full
trunk
1
1500
1
-
1-3967,4048-4093
-
SW2(config-if)# sh vpc
Legend:
vPC domain id
:
Peer status
:
:
Configuration consistency status:
:
:
vPC role
:
:
Peer Gateway
:
:
:
200
peer is alive
success
success
success
secondary
1
Disabled
Enabled

--------------------------------------------------------------------id
Port
Status Active vlans
---------- -------------------------------------------------1
Po23
up
1
181
vPC status
--------------------------------------------------------------------------id
Port
Status Consistency Reason
Active
vlans
------ ----------- ------ ----------- -------------------------- ---------102
Po102
down* failed
Compatibility check failed for native vlan
Like previously mentioned. The show vpc shows already most important information.
In a more recent NX-OS release, it became possible to have the vPC come online even when there is a
consistency error. The graceful consistency-check enables the interface of the vPC
terminating on the Primary vPC peer device to come online and have the interface terminating on the
Secondary peer go in suspended mode. This way the traffic will not have impact on a configuration
error.
In this example we used a Static Port-Channel instead of an LACP one.
You couldve used an LACP channel as well, as the question did not indicate which way to use. To
demonstrate that static port-channels also work in this configuration this was used in the example.
The next task is to use the remaining connections to finish the vPC configurations. These connections
that remain will be a vPC that is connected to another vPC. For the switches itself they will look like
standard vPC port-channels, but they are what is called back-to-back-vPCs.
This single connection is definitely a best practice. The important part of this is that the vPC Domain IDs
not never overlap in this configuration. This can lead to very strange situations.
The port-channel is configured just like any other vPC interface:
SW1-1
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
SW1-1(config-if)#
int e3/2,e3/4,e3/6
no shut
int port-channel103
sw mode trunk
vpc 103
no shut
SW1-2
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
SW1-2(config-if)#
int e3/5
no shut
int port-channel103
182

SW1-2(config-if)# vpc 103
SW1-2(config-if)#
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int e1/4
no shut
int port-channel103
sw mode trunk
vpc 103
no shut
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int e1/1-2,e1/4
no shut
int port-channel103
sw mode trunk
vpc 103
no shut
The next question in which we are asked to allow all VLANs required for our Layer 3 topology to be
traveling the vPC links is already completed as we did not place an VLAN allowed list on the vPCs. So we
can just use them for the next tasks.
The final task is to let the vPC domain 100 identify itself through LACP using a specific MAC address.
One reason for LACP port-channels to work is, that they ensure loop prevention. When different
switches are seen on interfaces the port-channel will not come online. This would cause issues on vPC
enabled switches as they still have 2 separate control planes and therefore different MAC addresses. To
be able for LACP port-channels to work. The vPC switch spoofs a single MAC address of one of the vPC
peer members for this.
The field we are looking for in the LACP packet is the System-MAC field. This is changed in the vPC
domain configuration.
SW1-1
SW1-1(config-vpc-domain)# system-mac 1234.5678.90ab
Changing system id might flap peer-link and vPCs. Continue (yes/no)? [no]
y
SW1-2
SW1-2(config-vpc-domain)# system-mac 1234.5678.90ab
Changing system id might flap peer-link and vPCs. Continue (yes/no)? [no]
y
183
As the command already mentions. The vPC peer-link and vPCs will flap when this command is issued
and you have LACP based port-channels running on the vPC enabled switch.
After this configuration task 3 is completed!
Task 4: Graceful Restart / Non-Stop Forwarding

The next task is regarding a feature which is by default configured in NX-OS based systems. The
technology has been around for a long time, but wasnt widely used, except in Service Provider
environments.
The Graceful Restart functionality is a process that works in dynamic routing protocols. It only
works on platforms that have a clear separation between control-plane and data-plane. The technology
requires the support of the dynamic routing protocols on which it is enabled. These protocols are able
to signal a special packet once the router switches over to another Supervisor (Route Processor, Route
Switch Processor, Routing Engine, etc.) or when they will completely reboot.
By signaling this special packet. The neighboring routers will know that a router is undergoing a change
(software upgrade, reboot, switchover, etc.). They will keep the routing information about this neighbor
in their respective routing tables and signal the neighbor down according to standard time-outs. The
traffic can still be directed towards the neighbor who is rebooting, because the data/forwarding plane is
not dependent on the control-plane to operate. Therefore traffic can still be sent to it.
When the router is rebooted / switched over, the dynamic routing protocol adjacencies are established
again. Routes are being learned using the normal process and the router is back up and running. Traffic
had no impact on this change.
The routing protocols that are supported on the NX-OS devices like OSPF and EIGRP are by default
enabled for graceful-restart. The feature will only work when both neighbors advertise the capability to
each other. So its safe to enable the feature, if the other neighbor doesnt support it, the feature is not
used.
In this task we first need to configure some dynamic routing protocols. We will use the routing
information configured now in later tasks!
First the OSPF configuration:
SW1-1
SW1-1(config)# router ospf IPX
SW1-1(config-router)# int vlan 112
SW1-1(config-if)# ip router ospf IPX area 0
SW1-1(config-if)# ip router ospf IPX area 0
184
SW2
SW2(config)# router ospf IPX
SW2(config-router)# int vlan 112
SW2(config-if)# ip router ospf IPX area 0
SW2(config-if)# ip router ospf IPX area 0
After configuration we can verify that switches can ping each others Loopback addresses.
198.18.0.2/32, ubest/mbest: 1/0
*via 198.18.112.2, Vlan112, [110/41], 00:00:06, ospf-IPX, intra
SW1-1(config-if)# ping 198.18.0.2 sour 198.18.0.11
SW1-1(config-if)#
Next up is the EIGRP configuration. In the question there is no mentioning about a EIGRP autonomoussystem number. So we can use any number we like in this case.
SW1-2
SW1-2(config-if)# feature eigrp
SW1-2(config)# router eigrp IPX
SW1-2(config-router)# int vlan 123
SW1-2(config-if)# ip router eigrp IPX
SW1-2(config-if)# ip router eigrp IPX
SW3
SW3(config-if)# feature eigrp
SW3(config)# router eigrp IPX
SW3(config-router)# autonomous-system 65000
SW3(config-router)# int vlan 123
SW3(config-if)# ip router eigrp IPX
SW3(config-if)# ip router eigrp IPX
185
After configuration we can verify that switches can ping each others Loopback addresses.
SW3(config-if)# sh ip eigrp nei
H
Address
Interface
Hold Uptime SRTT
Seq
(sec)
(ms)
Num
0
198.18.123.12
Vlan123
13
00:00:01 1
0
RTO
Q
Cnt
2000
SW3(config-if)#
SW3(config-if)# sh ip rout eigrp
198.18.0.12/32, ubest/mbest: 1/0
*via 198.18.123.12, Vlan123, [90/130816], 00:00:03, eigrp-IPX,
internal
SW3(config-if)# ping 198.18.0.12 sour 198.18.0.3
SW3(config-if)#
The next question in this task is already a default configuration, so this doesnt require any changes. It is
stated to make you aware that Graceful Restart or Non-Stop Forwarding is by default
configured on both EIGRP and OSPF configurations.
To verify you can issue a couple commands:
SW2(config-router)# sh ip ospf IPX
Routing Process IPX with ID 198.18.0.2 VRF default
Stateful High Availability enabled
Graceful-restart is configured
Grace period: 60 state: Inactive
Last graceful restart exit status: None
Supports only single TOS(TOS0) routes
Supports opaque LSA
Administrative distance 110
Reference Bandwidth is 40000 Mbps
Initial SPF schedule delay 200.000 msecs,
minimum inter SPF delay of 1000.000 msecs,
maximum inter SPF delay of 5000.000 msecs
186
Initial LSA generation delay 0.000 msecs,

minimum inter LSA delay of 5000.000 msecs,
maximum inter LSA delay of 5000.000 msecs
Minimum LSA arrival 1000.000 msec
LSA group pacing timer 10 secs
Maximum paths to destination 8
Number of external LSAs 0, checksum sum 0
Number of opaque AS LSAs 0, checksum sum 0
Number of areas is 1, 1 normal, 0 stub, 0 nssa
Number of active areas is 1, 1 normal, 0 stub, 0 nssa
Area BACKBONE(0.0.0.0)
Area has existed for 00:02:15
Interfaces in this area: 2 Active interfaces: 2
Passive interfaces: 1 Loopback interfaces: 1
No authentication available
SPF calculation has run 6 times
Last SPF ran for 0.000210s
Area ranges are
Number of LSAs: 3, checksum sum 0xe524
SW2(config-router)#
Next we configure timers in which we want to change the graceful-restart behavior. By default
the time-out of switches to reboot is set to 60 seconds. The question states that we will have a switch
connected to SW2 which takes longer to reboot. Therefore we need to change SW2 as that will be the
neighbor of the new switch.
SW2
SW2(config-if)# router ospf IPX
SW2(config-router)# graceful-restart grace-period 120
SW2(config-router)# sh ip ospf IPX | sec GracefulGraceful-restart is configured
Grace period: 120 state: Inactive
Last graceful restart exit status: None
The next configuration is a bit more tricky. It comes down to the fact that you need to have
Graceful-Restart enabled on the EIGRP process before an In-Service-Software-Upgrade
will work. You will be warned when you disable the graceful-restart option and start an ISSU.
We can verify that its by default enabled and that we are capable of supporting ISSUs!
SW1-2(config-if)# sh ip eigrp IPX
Process-tag: IPX
Status: running
Max paths: 8
187

SW1-2(config-if)#
In the EIGRP process we have a few different timers than are available in OSPF. In the EIGRP database
routes are maintained, but its already signaled to further EIGRP neighbors that one particular router in
the network is rebooting.
SW1-2(config-router)# timers nsf ?
converge
EIGRP time limit for convergence after switchover
route-hold EIGRP hold time for routes learned from nsf peer
signal
EIGRP time limit for signaling NSF restart
For this question we will change the hold-timer and the restart timer.
SW3
SW3(config-if)# router eigrp IPX
SW3(config-router)# timers nsf route-hold 300
SW3(config-router)# timers nsf signal ?
<10-360> Seconds
*Default value is 20
SW3(config-router)# timers nsf signal 10
We should configure the fastest restart signaling timer. By using the question mark we find that the
timer can be set to a value between 10 and 360. This means that 10 seconds is the shortest time period
in which we can signal a restarting neighbor to other EIGRP neighbors.
Next up are the First Hop Redundancy Protocols!
Task 5: HSRP
This is the first of three tasks regarding a subject which is a lot alike. HSRP or Hot Standby
Routing Protocol is one of the 3 major First Hop Redundancy Protocols (FHRP).
These protocols ensure that for any given Layer 2 segment the Default Gateway is protected against
failures.
Clients can only set one default gateway IP address in their Network Interface Card (NIC) settings.
This gateway should therefore always be up, otherwise clients are disconnected from the network as
soon as 1 router fails.
The solution to this problem is a protocol which enables resiliency between routers while still being able
to have only 1 IP address as the default gateway for a Layer 2 segment (IP subnet).
188
This is achieved by using a Virtual IP address (VIP) which is configured on 2 switches. One of
the 2 switches will be the active or primary router and the other switch is the backup or hotstandby. As soon as something happens with the primary switch, the next one will take over.
There are 3 major protocols that have similar functionality. The first that is going to be discussed is the
first protocol that was developed: HSRP.
The HSRP protocol is a Cisco proprietary FHRP. It uses a Virtual IP address connected to a Virtual MAC
address from the MAC address range of 0000.0C07.AC00 up to 0000.0C07.ACFF for HSRP
version 1 and 0000.0C9F.F000 up to 0000.0C9F.FFFF in HSRP version 2. Where the last
byte of the MAC address is related to the HSRP group number that is configured.
The working of HSRP is illustrated in the following drawing:
The primary HSRP router will be the only router that responds to ARP requests towards the Default
Gateway of the IP subnet. During the time both routers are active. The HSRP routers send Hello packets
to each other, informing them to be alive. Once a router fails and the hot-standby router reaches a
threshold of missing Hello packets it will become the new Primary router.
The selection of the primary router is done based on priority values that can be configured
(default 100). When the previously primary router comes back online, the HSRP protocol can, if
configured, fall back to the scenario where the highest priority router is the active router.
189
HSRP is available in 2 versions. Version 1 is by default enabled, version 2 can be configured

dependent on the requirements. These are the differences between the 2 versions:
Uses a dedicated Multicast group
HSRP version 2 uses 224.0.0.102 as its Multicast group to signal to the other router. HSRP version 1
used the All-Routers multicast group of 224.0.0.2
Group numbers can be configured from 0 to 4095
In HSRP version 1 this was limited from 0 to 255
Uses a different Virtual MAC address range
HSRP version 2 uses a different virtual MAC address range, to ensure that there are no compatibility
errors between the 2 protocols. They are completely different in terms of communication
Support for MD5 authentication
HSRP version 1 only supported clear-text authentication
Under normal conditions in an HSRP environment there is always only 1 router active on any given time.
In vPC configurations the Nexus switches can do a trick where they can both be active for an HSRP
group. This is required, otherwise the vPC peer-link would be required to carry a lot of traffic between
the 2 switches as all incoming traffic on the non-primary vPC peer would need to send its traffic to the
primary vPC peer to be layer 3 switched.
In vPC environments both routers will respond to the HSRP Virtual MAC address. This is called vPC
peer gateway which needs to be enabled. This requires that both vPC peers have an equal amount
of SVI or other routed interfaces to the Layer 3 network. Otherwise the vPC process cant ensure a
successful failover. Without the peer-gateway command, the vPC switches will not respond to their
configured Virtual MAC Address, which may lead to a lot of traffic crossing the peer-link.
The initial configuration of this task is to configure the basic parameters of Cisco HSRP. We will be using
the 198.18.111.1 IP address as the default gateway for this segment. After configuring the basic
parameter (the Virtual IP address). The protocol already works!
SW1-1
SW1-1(config)# feature hsrp
SW1-1(config-if)# hsrp 111
SW1-1(config-if-hsrp)# ip 198.18.111.1
SW1-2
190

SW1-2(config-if-hsrp)# ip 198.18.111.1
According to the MAC address tie-breaker we see that SW1-1 has become the Primary router for this
subnet and SW1-2 the hot-standby. For the group number we used the VLAN number. As there was no
indication in the question to use any specific group number.
SW1-1(config-if-hsrp)#
P
|
Interface
Grp Prio P
addr
Vlan111
111 100
198.18.111.1
(conf)
sh hsrp brief
indicates configured to preempt.
P
|
Interface
Grp Prio P
addr
Vlan111
111 100
198.18.111.1
(conf)
sh hsrp brief
State
Active addr
Standby addr
Active
local
198.18.111.12
State
Active addr
Standby addr
Standby
198.18.111.11
local
Group
Group
The next question will decide which router will be the primary router in VLAN 111.
The question states that this should follow the best practice. If we take a look at the configuration guide
about this point. We see that Cisco is recommending to make the router HSRP primary if it also has
the vPC Primary role assigned. In our case its SW1-2 that is the primary vPC device in our network.
So we will make this the primary router in our HSRP network too.
There are no priority values given for this task so we can use anything we like. However, when you read
ahead of the questions. You will find a question related to Interface Tracking. These questions
ask you to subtract some value from the priority. If we use numbers now, it can help us later with these
questions.
We will use simple numbers for the priority values. I always keep as a best practice to make the standby
switch lower than the default value and the primary switch higher than the default value of the HSRP
priority, which is 100.
SW1-1
SW1-1(config-if-hsrp)# priority 90
SW1-2
191

SW1-2(config-if)# priority 110
After configuring the priority values and flapping the interface or the HSRP group. We see that SW1-2 is
now the active router.
SW1-1(config-if)# sh hsrp brief
P indicates configured to preempt.
|
Interface
Grp Prio P State
Active addr
Standby addr
addr
Vlan111
111 90
Standby 198.18.111.12
local
198.18.111.1
(conf)
SW1-1(config-if)#
P
|
Interface
Grp Prio P
addr
Vlan111
111 110
198.18.111.1
(conf)
Group
sh hsrp brief
State
Active addr
Standby addr
Active
local
198.18.111.11
Group
Next is the security of the HSRP protocol. In the question it is asked to use an hashed key. Meaning that
we will be using MD5 authentication. This also immediately implies that we need to use HSRP
version 2, because we wouldnt have the option to configure MD5 otherwise.
Its a very dangerous configuration, because MD5 is perfectly configurable without HSRP version 2,
but the protocol keeps functioning without any authentication applied then!
Even show commands do not show you what is happening at that moment, but the traffic is not
authenticated without specifying the HSRP version 2.
In the question the authentication is also given a specific schedule. This means we will need to be using
a key-chain to support this. Key chains are a mechanism to switch from authentication keys in a
given time schedule. This ensures security of the network and the exchange of keys. The timelines are
configurable and its also possible to accept an older key longer to ensure a smooth migration to the
new key.
SW1-1
SW1-1(config)# key chain HSRP
SW1-1(config-keychain-key)# key-string IPexpertYEAR1
SW1-1(config-keychain-key)# send-lifetime 12:00:00 Oct 4 2012 23:59:59 Dec
31 2012
SW1-1(config-keychain-key)# accept-life 12:00:00 Oct 4 2012 02:00:00 Jan
01 2013
192
SW1-1(config-keychain-key)# key 2
SW1-1(config-keychain-key)# send-life 00:00:00 Jan 01 2013 infinite
SW1-1(config-keychain-key)# accept-life 00:00:00 Jan 01 2013 infinite
SW1-1(config-keychain-key)# interface vlan 111
SW1-1(config-if-hsrp)# authentication ?
WORD Plain text authentication string (Max Size 8)
md5
Use MD5 authentication
text Plain text authentication
SW1-1(config-if-hsrp)# authentication md5 ?
key-chain
Set key chain
key-string Set key string
SW1-1(config-if-hsrp)# authentication md5 key-chain HSRP
SW1-1(config-if-hsrp)# exit
SW1-1(config-if)# hsrp ?
<0-4095> Group number
bfd
BFD protocol
delay
HSRP initialisation delay
use-bia
HSRP uses interface's burned in address
version
HSRP version
SW1-1(config-if)# hsrp version 2
SW1-1(config-if)#
The key chain configuration is done by using a number of keys. By using a send-lifetime and
accept-lifetime its configurable how long the key is sent and how long it is accepted as a valid
key. The initial key that is configured is sent from the current date and time up until December 31st. As
the question stated the key should be accepted for another 2 hours, to prevent time synchronization
issues between routers, which could cause an outage when authentication is rejected. The next (new)
key that is configured has an accept and send lifetime of infinite, because the question did not state
when the key should be changed again. Therefore we just use it forever.
The key chain is then configured on the HSRP configuration. Like previously mentioned, this is not
enough. HSRP will not use the MD5 configured authentication parameters when version 1 is used.
The HSRP version is not configured under the regular HSRP process configuration, but under the
interface itself. Version 2 is required for MD5 authentication. After this has been changed the HSRP
group uses the authentication parameters and authenticates the hello messages that are send and
received.
SW1-2
SW1-2(config)# key chain HSRP
SW1-2(config-keychain-key)# send-lifetime 12:00:00 Oct 4 2012 23:59:59 Dec
31 2012
SW1-2(config-keychain-key)# accept-life 12:00:00 Oct 4 2012 02:00:00 Jan
01 2013
SW1-2(config-keychain-key)# key 2
193

SW1-2(config-keychain-key)# send-life 00:00:00 Jan 01 2013 infinite
SW1-2(config-keychain-key)# accept-life 00:00:00 Jan 01 2013 infinite
SW1-2(config-keychain-key)# interface vlan 111
SW1-2(config-if-hsrp)# authentication md5 key-chain HSRP
SW1-2(config-if)# hsrp version 2
SW1-2(config-if)#
After both switches have been configured, the authentication is now used for the HSRP process.
SW1-2# sh hsrp brief
P indicates configured to preempt.
|
Grp Prio P State
Active addr
Standby addr
Interface
Group
addr
Vlan111
111 110
Active
local
198.18.111.11
198.18.111.1
(conf)
SW1-2#
SW1-2# sh hsrp
Vlan111 - Group 111 (HSRP-V2) (IPv4)
Local state is Active, priority 110 (Cfged 110)
Forwarding threshold(for vPC), lower: 1 upper: 110
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.163000 sec(s)
Virtual IP address is 198.18.111.1 (Cfged)
Active router is local
Standby router is 198.18.111.11 , priority 90 expires in 5.503000 sec(s)
Authentication MD5, key-chain HSRP
Virtual mac address is 0000.0c9f.f06f (Default MAC)
5 state changes, last state change 01:38:49
IP redundancy name is hsrp-Vlan111-111 (default)
SW1-2#
The following question relates to a feature called pre-emption. This feature ensures that when a
router comes online in an HSRP network and it has a higher priority configured it will assume the active
role. By default the current active router will keep its role when a higher priority router is seen. With
preempt enabled the router with the higher priority value will always take over (take back) its active
role.
The take over process can be delayed to ensure that the router that comes online (which is usually after
a reboot), is delayed before taking back its active role to have all other protocol running on this router
to synchronize correctly before it will attract traffic.
The question states that the active router has been rebooted and then comes online. This means that
the delay is configured for a reload to ensure all protocols have synchronized.
SW1-1
SW1-1(config-if-hsrp)# int vlan 111
194

SW1-1(config-if-hsrp)# preempt delay reload 180
SW1-2
SW1-2(config-if-hsrp)# preempt delay reload 180
The next question is related to giving the HSRP process a name, which is an easy knob in the HSRP
configuration.
SW1-1
SW1-1(config-if-hsrp)# name IPexpertVLAN111
SW1-2
SW1-2(config-if-hsrp)# name IPexpertVLAN111
Next are changing timers. With the default timers its only possible to configure up to a hellointerval of 1 second. With the special knob msec its possible to configure sub-second hellotimers and therefore a convergence within 1 second
SW1-1
SW1-1(config-if-hsrp)# timers msec 300 msec 1000
SW1-2
SW1-2(config-if-hsrp)# timers msec 300 msec 1000
The next 2 questions are related to HSRP object tracking. This tracking ensures that when the tracking
group fails, the HSRP priority is lowered. In legacy scenarios when the priority falls under the priority of
the hot-standby router and pre-emption is configured. The hot-standby takes over the active role.
195
In this scenario on NX-OS with vPCs on the switch. The failure of a switch will result in a failover to the
hot-standby router of course. But there is no single distinction anymore between the active and hotstandby router. Both routers are forwarding traffic. What can be configured by using the priority
command is to configure thresholds when the traffic forwarding should be stopped and traffic is no
longer layer 3 routed on the NX-OS device, but rather send across the vPC Peer-link towards the vPC
peer for layer 3 routing.
In our case of the questions, it should take 2 uplinks to fail on the active switch before it stops
forwarding traffic on Layer 3 and sending traffic on a Layer 2 basis towards the vPC peer.
Currently we have a priority of 110 configured on the primary switch (or any other number that you
configured in a previous question). 10% of this value should be subtracted once an uplink fails. When a
second interface fails it should stop layer 3 forwarding completely.
10% of the 110 value is 11. So we subtract 11 from each of the failing Ethernet uplinks. Then we
configure the thresholds for forwarding appropriately. Remember to use a different value for the hotstandby switch as 10% from 90 is only 9 and not 11.
Depending on the values you chose at the earlier question. You need to ensure that when you subtract 2
times 10% from that value, it should be lower than the hot-standby switch, so it will assume the active
role after 2 uplinks have failed.
In our case the priority value will drop with 22 (2 times 11) and therefore be 88. Therefore the hotstandby switch will take over as its configured for priority 90.
Remember that pre-emption needs to be configured for a fail-over when priority drops
SW1-1
SW1-1(config-if-hsrp)# priority 90 forwarding-threshold lower 72 upper 81
SW1-1(config)# track 1 ?
<CR>
interface Interface to track
ip
IPv4 protocol
ipv6
IPv6 protocol
list
Object tracking list
SW1-1(config)# track 1 interface ethernet3/3 ?
.
Sub interface separator
ip
IPv4 parameters
ipv6
IPv6 parameters
line-protocol Track interface line-protocol
SW1-1(config)# track 1 interface ethernet3/3 line-protocol
SW1-1(config-track)# exit
SW1-1(config)# track 2 interface e3/5 line-protocol
196
SW1-1(config-if-hsrp)# track 1 decrement 9
SW1-2
SW1-2(config)# track 1 interface ethernet3/7 line-protocol
SW1-2(config)# track 2 interface e3/9 line-protocol
SW1-2(config-if-hsrp)# priority 110 forwarding-threshold lower 88 upper 99
Interface tracking can be done using multiple configuration options. The question now states that we
need to track an interface for going down. This means we do not need to do anything with the IP related
tracking features for now. We can just track on line-protocol going down and the tracking group will go
down.
The final question of this task is to use a physical interface MAC address instead of a virtual
MAC. Some systems might have trouble seeing special MAC addresses like the one used by HSRP.
Therefore there is a fallback to a physical MAC address to use as Virtual. This feature will borrow a
MAC address from the physical interface or SVI from the Active router. When the active router fails its
MAC address is then moved just like a regular HSRP action to the hot-standby router. The MAC address
will stay the same even though it does not belong to the new active router.
The configuration is fairly easy, although not specified under the HSRP configuration, but under the
interface directly.
SW1-1
SW1-1(config-if)# hsrp use-bia
SW1-2
SW1-2(config-if)# hsrp use-bia
If we now check the HSRP group, we see that its using a real MAC address as its virtual MAC address.
SW1-1(config-if)# sh hsrp
Vlan111 - Group 111 (HSRP-V2) (IPv4)
Local state is Active, priority 72 (Cfged 90)
Forwarding threshold(for vPC), lower: 72 upper: 81
197
Hellotime 300 msec, holdtime 1 sec

Next hello sent in 0.255000 sec(s)
Virtual IP address is 198.18.111.1 (Cfged)
Active router is local
Standby router is 198.18.111.12 , priority 110 expires in 0.915000
sec(s)
Authentication MD5, key-chain HSRP
Virtual mac address is 0024.98e8.01c4 (Interface MAC - use-bia enabled)
2 state changes, last state change 01:21:04
Track object 1 state DOWN decrement 9
Track object 2 state DOWN decrement 9
IP redundancy name is IPexpertVLAN111 (cfgd)
SW1-1(config-if)#
Next task is about the standards based solution based on the Cisco proprietary HSRP protocol!
Task 6: VRRP
The second FHRP protocol that is discussed is the Virtual Router Redundancy Protocol (VRRP). This
protocol is very similar in operation to Ciscos HSRP. It borrows a lot of technical specifications. This
protocol is an open standard based on RFC 3768 (version 2) or RFC 5798 (version 3).
All routing vendors have implementations for the VRRP. This makes it the best practice protocol to use
when you are using a mixed-vendor environment. Cisco is not active in implementing new features in
VRRP compared to HSRP. A lot of advanced features like MD5 authentication or IPv6 support are not
available in latest releases of NX-OS. Version 3 of VRRP (which brings IPv6 support) is not available in
NX-OS yet. Other features like BFD tracking, which have long been available in HSRP, are only recently
implemented for VRRP.
The questions in this task are very similar to the questions in the HSRP task as the protocol works almost
the same as HSRP.
The first few questions are related to configuring the basic configuration parameters of the VRRP. The
third question however is asking for a special configuration. A special virtual MAC address should be
used for the VRRP group.
VRRP uses MAC address from the range of 0000.5E00.0100 to 0000.5E00.01FF. We are asked
to configure a virtual MAC address of 0000.5E00.0174. This means we should use a special group
number. As the group number is the last byte of the MAC address, just like with HSRP.
When you convert the number 74 from hexadecimal to decimal we do the following steps:
We first convert the hex number 7 to binary:
0111
Next is converting the number 4 to binary:
198
0100
A hex character consists of 2 nibbles making up 1 byte. We put the 2 nibbles together and convert the
binary byte to a decimal number:
01110100 = 116
This means we will be using VRRP group number 116.
SW1-1
SW1-1(config-if)# vrrp 116
SW1-1(config-if-vrrp)# address 198.18.121.254
SW1-2
SW1-2(config-if-vrrp)# address 198.18.121.254
SW1-2(config-if-vrrp)# priority 200
After configuration, it seems that the VRRP group is not working correctly.
SW1-2(config-if)# sh vrrp
Interface VR IpVersion Pri
Time Pre State
VR IP addr
--------------------------------------------------------------Vlan121 116
IPV4
200
1 s Y
Init 198.18.121.254
This is a very strange message as the reason why its staying on Init is very clear when you take a
closer look.
SW1-2(config-if)# sh vrrp detail
Vlan121 - Group 116 (IPV4)
State is Init(Administratively down)
Virtual IP address is 198.18.121.254
Priority 200, Configured 200
Forwarding threshold(for VPC), lower: 1 upper: 200
Advertisement interval 1
Preemption enabled
Virtual MAC address is 0000.5e00.0174
Master router is Unknown
Its clearly seen that the VRRP group is by default on a shutdown state and it needs to be enabled
before it will start sending VRRP packets.
We can immediately see that the virtual MAC address is as it should be per the question. This means we
did our calculation correctly.
199
SW1-1
SW1-1(config-if-vrrp)# no shutdown
SW1-2
SW1-2(config-if-vrrp)# no shutdown
After a no shutdown the VRRP group comes online!

SW1-2(config-if-vrrp)# sh vrrp
Time Pre State
VR IP addr
--------------------------------------------------------------Vlan121 116
IPV4
200
1 s Y Master 198.18.121.254
The next task asks to configure authentication on the VRRP group. MD5 authentication is not supported
in NX-OS at the time of writing. Authentication is therefore easily configured.
SW1-1
SW1-1(config-if-vrrp)# authentication text IPexpert
SW1-2
SW1-2(config-if-vrrp)# authentication text IPexpert
Now we verify we indeed see that the process is authenticated on the routers
SW1-2(config-if-vrrp)# sh vrrp detail
Vlan121 - Group 116 (IPV4)
State is Master
Virtual IP address is 198.18.121.254
Priority 200, Configured 200
Forwarding threshold(for VPC), lower: 1 upper: 200
Advertisement interval 1
Preemption enabled
Authentication text "IPexpert"
Virtual MAC address is 0000.5e00.0174
Master router is Local
One thing different from the Cisco HSRP is that pre-emption is enabled by default. Again this ensures
that when a higher priority router comes online on the subnet, the routers will take over the active role
pro-actively. In the question is stated that we dont want this to happen, so we should disable it.
200
We are asked to only configure the Active switch. This means that an initial failover will happen right
away due to pre-emption, but a fall-back will only occur when the other switch fails.
SW1-2
SW1-2(config-if-vrrp)# no preempt
The next question is a little more tricky. VRRP does not support the tracking of Layer 2 interfaces. It does
support direct tracking of Layer 3 interfaces, where no tracking groups are necessary. Instead of tracking
an interface in this question we are tracking the reachability of a route in the routing table.
This means that as soon as SW3 fails, the Loopback address will disappear from the routing table on
SW1-2. This as per our question means we need to do a failover to the other VRRP router. This process
should be delayed with 30 seconds. Remember to not override your previous tracking group
configuration, otherwise you will lose points in the HSRP task!
SW1-2
SW1-2(config)# track 3 ip route 198.18.0.3/32 reachability
SW1-2(config-track)# ?
delay Tracking delay
no
vrf
Configure VPN Routing/Forwarding table
end
Go to exec mode
exit
pop
push
where Shows the cli context you are in
SW1-2(config-track)# delay ?
down Delay down change notification
up
Delay up change notification
SW1-2(config-track)# delay down 30
There is no specification how much the group should be decremented. Therefore we just ensure that
the priority value is decreased under the default of 100. This would cause the switch to take over the
active role as we did not disable pre-emption on the other switch.
SW1-2
SW1-2(config-if-vrrp)# track 1 decrement 101
The final question of this task is to configure timers of VRRP. The only trick you have to change timers is
to change the interval in which VRRP packets are advertised to all VRRP running routers on the
subnet. The holddown timer is not adjustable. This is by default 3 times the hello-interval including
201
some priority value timing. For any given configured priority value setting the timers are adjusted a little
bit, which is the way how VRRP elects its active and standby router.
If we want to reach a down signaling within 10 seconds, we will use a hello-interval of 3 seconds. When
3 hello packets are missed after 9 seconds including a little priority timing we will signal a VRRP
neighbor down within 10 seconds.
SW1-1
SW1-1(config-if-vrrp)# advertisement-interval 3
SW1-2
SW1-2(config-if-vrrp)# advertisement-interval 3
Now the VRRP task is finished. The final task for FHRP protocols is another Cisco proprietary protocol
supporting Load Balancing.
Task 7: GLBP
The last but not least FHRP that is tested in our workbook is another Cisco proprietary protocol called
the Gateway LoadBalancing Protocol (GLBP). This protocol is the most recent one of all
FHRPs. It uses a little different approach than HSRP and VRRP. The other FHRPs are using a
single active router in normal environments. In vPC environments this is a little different as
both HSRP and VRRP gateways are active and responding to client requests. The GLBP protocol
supports this by default, but its also possible to configure the load balancing , which is not possible in
case of the vPC load balancing trick.
Routers are given specific roles in the GLBP configuration:
Active Virtual Forwarder (AVF)
There can be multiple AVF routers in the GLBP network configuration
These are routers that are forwarding Layer 3 traffic according to the weight that is configured on them
to allow the traffic coming in.
AVFs have their own dedicated Virtual MAC Address for each separate AVF
Active Virtual Gateway (AVG)
202
This is only a single router in the GLBP configuration which is selected based on priority values.
This router will be responding to all ARP requests of the subnet.
The Active Virtual Gateway is aware of all Virtual MAC Addresses in the GLBP configuration and uses a
Round Robin, Weighted or Host Dependent mechanism to answer the ARP requests. This
means that each ARP request can have a different MAC address in the ARP reply. This is how the load
balancing is achieved.
The failover of an AVF is handled using 2 timers. The Redirect Timer is used to gracefully take out
the Virtual MAC address of the AVF of the Virtual MAC Pool maintained by the AVG. The Secondary
Hold Timer is used when a AVF has completely failed and the Virtual MAC Address of this AVF will be
freed in the pool and is available for possible new AVF routers.
There are 3 ways of answering the ARP requests on the AVG and achieving load balancing. These are
explained in the following table.
Mode
Description
Round Robin
This is the most granular way of load balancing as the AVG answers a other
Virtual MAC Address on each ARP request. Meaning traffic is equally load
balanced across all AVFs
Weighted
This mode enables the use of weight values on AVFs. When a higher weight
is advertised by an AVF. The AVG will answer more ARP requests with this
Virtual MAC Address so this AVF will handle more traffic than others
Host dependent
The host dependent mode guarantees that the same host receives the
same AVF each time it does an ARP request.
You can change the weight or priority values for becoming an AVF or AVG according to interface
tracking, just like in the other FHRPs.
In the current software releases used for this version of the CCIE Datacenter test the GLBP is only
supported on the Nexus 7000 series switches. Additionally there is currently no support for IPv6 in the
protocol.
203
The GLBP protocol is illustrated using the following drawing.
Now to start our configuration we begin with fairly basic tasks of configuring the GLBP on VLAN 222
on our Nexus 7000 series switches.
SW1-1
SW1-1(config-if)# feature glbp
SW1-1(config-if)# glbp 222
SW1-1(config-if-glbp)# ip 198.18.222.55
SW1-1(config-if-glbp)# no shutdown
SW1-2
SW1-2(config-if)# feature glbp
SW1-2(config-if-glbp)# ip 198.18.222.55
SW1-2(config-if-glbp)# no shutdown
The next 2 tasks first require to assign the AVG role to SW1-1 and that both routers should be AVF. By
default all GLBP capable routers are used for load balancing traffic to. A router can decide for itself if it
does not want to participate in the load balancing anymore, by configuring upper and lower limits. We
will get there in a later question in this task.
SW1-1
SW1-1(config-if-glbp)# priority 110
204
The next question is related to the previous one. We need to ensure that both routers are having an AVF
role in the network, but do want to remove the router as an AVF when the Loopback address of the
upstream switches disappears from the routing table.
Just like in the VRRP task we can use IP Route Reachability tracking for this. We will configure
lower thresholds when the router will give up its AVF role. Again do not forget to use unique numbers
for the tracking groups. Otherwise you will overwrite configuration from a previous task.
SW1-1
SW1-1(config-if-glbp)# track 1 decrement 20
SW1-1(config-if-glbp)# weighting 110 lower 90 upper 110
SW1-2
SW1-2(config-if-glbp)# track 1 decrement 20
SW1-2(config-if-glbp)# weighting 110 lower 95 upper 105
With this configuration we will take out the router from the AVF role (and the AVG from its role when
applicable) when the reachability towards a IP prefix is gone.
This process should be delayed. We can do this in 2 ways. The first is what we did in the VRRP task, by
delaying the tracking group for signaling the loss of reachability. If you read the question carefully it
mentions that you need to support yourself for a failure of a current AVF. The AVG will only take out a
failed AVF from the pool after the redirect timer has expired, after which it will keep its Virtual
MAC reserved for another timer (secondary timeout timer).
When we change the redirect timer the AVG will take out the AVF and that is what this question is
asking for. Keep in mind to change both switches as they both might become the AVG at some point.
SW1-1
SW1-1(config-if-glbp)# timers redirect 180
SW1-2
SW1-2(config-if-glbp)# timers redirect 180
205
The next task is regarding the pre-emption of the AVG role. The preempt command is disabled by
default in the GLBP protocol. By default the delay for taking over from a lower priority router is set to
3600 seconds. This is significantly longer than any other timer in a FHRP. The reason for this is that a
router immediately becomes an AVF when it comes online, so its already forwarding traffic. The
switchover of the AVG role is therefore not that important anymore.
SW1-1
SW1-1(config-if-glbp)# preempt delay minimum 30
SW1-2
SW1-2(config-if-glbp)# preempt delay minimum 30
The final task is regarding ISSUs on GLBP enabled routers. The GLBP protocol supports a nice feature for
supporting Non Stop Forwarding. You can set timers as short as you like, but when a restart is
done of any neighbor or the router itself, it can temporarily extend the timers for the duration of
the Graceful Restart.
This is enabled with a single command under the global configuration:
SW1-1
SW1-1(config)# glbp timers extended-hold
SW1-2
SW1-2(config)# glbp timers extended-hold
And all FHRP tasks are successfully completed!
Task 8: Virtual Port-Channels (vPCs) and FabricPath

The last task of this High Availability chapter will be about a special feature for using vPCs in a FabricPath
enabled network. This is due to the architecture of the FabricPath protocol that it is not possible to
configure by default.
The FabricPath network uses Equal Cost MultiPathing to reach any given destination (MAC
address) in the network. Now when any FabricPath enabled router will perform a lookup on the nexthop it will result in a Destination Switch ID. For vPC connected hosts or switches this is
different as they are connected to 2 different FabricPath Switch IDs. It will result in strange
behavior and non-deterministic traffic flows throughout the FabricPath network.
206
The following drawing illustrates the problem in more detail:
The drawing illustrates that each packet, depending on the Port-channel hashing can be sent from or to
either S1 or S2. Everytime S3 learns the MAC address of the PC connected it will update its routing
table accordingly. This introduces a MAC address flapping issue on S3.
The solution that Cisco came up with has been called vPC+. This means that the vPC switches are
compatible with FabricPath enabled networks.
The solution is that the whole vPC domain will be virtualized as a single FabricPath Switch ID. In that way
the entry for the MAC address of the PC connected has a Destination Switch ID which is the virtual ID
configured statically on the vPC domain. That way the other FabricPath switches can still use Equal Cost
Multi-Pathing towards the host just like any other destination in the FabricPath network without causing
any loops.
The configuration is fairly simple. The task asks you to configure the FabricPath network first after which
it should be made ready for the vPC configuration.
As there is no host connected to the switches, you just need to prepare the configuration for it. After
this pre-configuration there is nothing different from a standard vPC configuration.
207
The topology that will be used for this lab is laid out in Drawing 3 in the workbook and for your
convenience showed below:
Now on to the configuration. We enable the FabricPath network on all interfaces.

SW2
SW2(config)#
SW3
SW3(config)#
208
SW1-1
SW1-1(config)#
SW1-2
SW1-2(config)#
Now to finish the chapter we need to configure the static FabricPath Switch ID under the vPC domain
configuration.
SW2
SW2(config)#
SW2(config-vpc-domain)# fabricpath switch-id 100
SW3
SW3(config)#
SW3(config-vpc-domain)# fabricpath switch-id 100
You finished Chapter 4 successfully! We also finished all Networking chapters at this moment and we
will continue with the Fibre-Channel features of both the Cisco MDS switches and the Nexus 5000
switches.
209
Chapter 5: Data
Center Storage
Networking
Chapter 5: Data Center Storage networking is intended to let you be familiar with the Storage
Networking features on the Cisco MDS switches. Configuring traditional Fibre Channel networks and
basic Fibre Channel features.
210
General Rules
Take your time. This is not a Mock Lab, so no time constraints are in place for finishing this particular
chapter
5 hours
211
Solutions
Task 1: Initial set-up
The first task of this storage networking chapter is to set-up the initial topology. We first configure
hostnames for the storage equipment and we configure management IP addressing. Our topology in this
exercise is fairly simple. This is because most of the storage networking (Fibre Channel) features
are tested in a separate CCIE Storage track. Therefore the CCIE Data Center doesnt have a lot of focus
on the SAN part.
However, the Fibre Channel world is completely different from the Ethernet world and therefore we
require some extensive configuration to have a good understanding of the Fibre Channel protocol and
the many differences that there are with Ethernet switching.
The Fibre Channel protocol has been around for a long time already and is used widely in the world. It
makes it possible for servers to extend the SCSI commands to hard disks over a data network. This is the
initial purpose of the Fibre Channel network. Fibre Channel works a little different in the OSI model than
Ethernet works.
You can say that the Fibre Channel protocol is much more intelligent than Ethernet. Most of the routing
protocols that we need to implement in the Ethernet and IP world are by default enabled in the Fibre
Channel world and do not require any configuration to work properly. Most of the protocols can come
online by default and are automatically converging fine.
The Fibre Channel protocol uses a special ID of disks and hosts (or targets and initiators) that identify
the host towards the network. These IDs are called World Wide Names (WWNs). A device always has
2 of these IDs, these are a World Wide Port Name (WWPN) and a World Wide Node Name
(WWNN). It makes sense that the WWPN is assigned for every port on the device and the WWNN is assigned
to a single device. A special WWN is assigned to every MDS switch. These are called Switch World
Wide Names (SWWNs). These IDs are used to identify any MDS switch in a topology and can be used
to reference a switch. The ID is also used by the switch in many of the control-plane protocols that are
running in the FC control-plane.
These IDs are most comparable to MAC addresses in the Ethernet world, except that the WWNs are
never used in the data communication! This is a common misconception that, like in Ethernet, the
physical addresses are used as Layer 2 protocol. They are just for identifying the host or port towards
the Fibre Channel environment.
After the identification the FC switch will hand out the logical addressing (like IP addresses in the
Ethernet/IP world). These addresses are called FCIDs. They are comprised of a 3-byte / 24-bit
number represented as 6 hexadecimal characters: 0x890ABC. The first byte of the FCID is the so-called
FC Domain ID of the MDS switch. The second byte is the area code and the third byte is the
address of the host. The Domain ID is an ID used by the Fibre Channel environment between FC
switches for routing!
212
The first byte of an FCID is used and a routing lookup is done against it to determine the destination FC
switch in the network. When the destination switch is reached, the area code is used to determine
the exit interface of this packet. Usually in an FC environment a whole area code is handed out to one
interface on the switch. After that each host/disk that is connected behind that interface, in case its an
NPV port or a Loop port, different devices get an ID from that area handed out.
The model looks a bit like a DHCP environment, except its distributed and each FC switch is participating
in this handing out of addresses. There is one switch in the network responsible for handing out
Domain IDs to FC switches and this is the Principal switch. This Principal switch is chosen
according to the highest priority number of as a tie-breaker the highest SWWN.
This model is supported by some control-plane protocols that each host on the network needs to
perform when it is connected to a switch. This process is called an FLOGI or Fabric Login process.
After the FLOGI a Port Login or PLOGI is done to identify all capabilities of the just connected
host.
In the configuration tasks more is explained about the Fibre Channel protocol.
The topology we use in this lab is simple. We have 2 native MDS switches. In a later chapter we will
include the Nexus switches in the Fibre Channel topology as well, but we are currently only focusing on
the native Fibre Channel topologies.
213
Connected to the 2 MDS switches are 2 JBODS. These devices are nothing more than containers for SCSI
disks that are capable of logging in to a Fibre Channel switch. Fibre Channel attached servers are then
able to connect to this physical hard-disk through the Fibre Channel network.
In the real world these disks would communicate to a Storage Filer where RAID configurations and
other access methods are applied.
Our topology will use the disks as dual-attached devices where we can simulate having a lot of devices
on the Fibre Channel network.
During this chapter the detailed solution guide will explain Fibre Channel (FC) concepts. This will not be
an in-depth explanation, but a basic explanation to ensure you understand the concept and how its
related to our CCIE Data Center topology.
The MDS switches will boot up with an empty configuration where we need to apply all configuration.
The MDS switches run the same code as the Nexus switches. The old SAN-OS has been rebranded to
NX-OS since version 4.0. The CLI is exactly the same, except that the FC protocol all has their own
configuration parameters which are not compatible with their Ethernet equivalents.
Initially we need to give those hostnames and configure the mgmt0 management Ethernet interfaces.
MDS1
Invalid admin password. Please try again.
Enter the password for "admin": ipexpert
Confirm the password for "admin":ipexpert
SETUP: Admin password config failed.
Password is not strong enough:it is should be atleast 8 characters
NX-OS enforces you to use a strong password!

Error(s) while applying config.
---- Basic System Configuration Dialog ---This setup utility will guide you through the basic configuration of
of the system.
Please register Cisco MDS 9000 Family devices promptly with your
service calls. MDS devices must be registered to receive entitled
support services.
214
MDS Switch
10.2.8.74 login: admin
Password: IPexpert123
Cisco Storage Area Networking Operating System (SAN-OS) Software
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software may be covered under the GNU Public
License or the GNU Lesser General Public License. A copy of
each such license is available at
http://www.gnu.org/licenses/gpl.html and
http://www.gnu.org/licenses/lgpl.html
switch# conf t
switch(config)# switchname MDS1
MDS1(config)# interface mgmt0
MDS1(config-if)# ip address 172.16.100.10/24
MDS1(config-if)# no shutdown
MDS1(config-if)# ip default-gateway 172.16.100.254
MDS1(config)# ntp server 172.16.100.254
MDS2
switch(config)#
switch(config)# switchname MDS2
MDS2(config)# interface mgmt0
MDS2(config-if)# ip address 172.16.100.11/24
MDS2(config-if)# ip default-gateway 172.16.100.254
MDS2(config)# ntp server 172.16.100.254
Verify that time is synchronized. Keep in mind that this might take up to 5 minutes before a full
synchronization is completed.
The lab states that we shouldnt use any automatic selection for interface speeds. This means that we
need to specify static link speeds on every link that connects a host to our MDS switches. Besides that
we shouldnt use a feature of the MDS switches that lets the switch automatically select the type of
interface that should be selected according to the connected device.
The type of interface is a crucial point in configuring interfaces in a Fibre Channel network. This changes
the behavior of the interface and the FC protocol. The Cisco MDS switches have a feature that they can
auto detect what kind of interface is connected on the other side and set the interface type according to
that.
215
The following table shows all the various interface types, which are configurable on the MDS switch and
what they do:
Mode
Description
The E-port is a so-called Fabric Extension port. This port ensures the extension of the
Fibre Channel fabric. A connection between 2 switches is always configured as an Eport. Pay attention to oversubscription on E-ports. You might need more buffer
credits on these ports as a lot of traffic might go across them.
TE
The TE-port is equal to the E-port in positioning, except that this port has the
Trunking protocol enabled, which enables the use of VSANs between Cisco MDS
switches.
The Fabric Port is the most common interface. You can attach Hosts and JBODs to
these interfaces. On the Host or JBOD itself its called an N-port. So an N-port always
connects to an F-port.
FL
A Fabric Loop port is used to connect Public Arbitrated Loops to the MDS switch.
Arbitrated Loops are self-supporting FC rings without switches between devices. This
is a way to connect them to an FC infrastructure
Fx
The Fx port can be either an F or an FL port. It will automatically detect if its a normal
N-port connected behind of it or an Arbitrated Loop. This mode can be the default
mode on switches.
TF
The TF port is equal to an F-port, but supports the Trunking protocol, to connect
VSANs towards hosts or NPV switches
NP
NP ports are equal to N-ports on a device, but these ports are on FC switches that are
set to N-Port Virtualization mode.
TNP
These ports are equal to NP ports, but support the Trunking (EISL) protocol, to
transfer VSAN information to NPV switches
SD
SD is the SPAN Destination port where a FC analyzer can be connected, like the Cisco
FC to Ethernet converter. This traffic receives traffic from other ports for analyzation
ST
SPAN Tunnel ports are used as reflector ports for Remote SPAN sessions. Traffic that
needs to be copied to another device is using this tunnel port for encapsulation.
216
Mode
Description
TL
This port connects Translative Loops to the MDS switches. It enables communication
between TL, FL and other devices in the FC fabric
Auto
This is the default mode on most MDS switches. This mode auto detects the mode in
which it needs to operate in. It can become a E, F, FL, TE or TF port. Other port modes
need to be manually configured.
In this exercise we shouldnt use the automatic interface type selection so we need to configure it
statically. This is so you practice with the various port types that we need.
In the CCIE Data Center lab its typical that you will be stated what kind of JBODs are attached to the
switches. These can be anything, from an N port to any kind of Private or Public loop. Be very careful as
you need to select the correct interface type, because when this is not properly configured you will not
see any FLOGIs from the attached devices.
There are a few more requirements that we need to comply with. We need to use 200MBps
connections towards the JBODs. Pay attention to the capital B in this word. Fibre Channel can be
configured to work in 1Gbps, 2Gbps, 4Gbps or 8Gbps speeds. However, the protocol uses an
8B/10B encoding mechanism, which means that for every 8 bits of data, 10 bits are transferred over
the link. This comes down to a perfect match in throughput that 1Gbps FC traffic is 100MBps of data
traffic and so on.
Now we start configuring the interfaces. You might need to detect how the JBODs are connected. Just
try a few interface types until you see FLOGIs coming in.
In our example the JBODs are using Public Arbitrated Loops which means we need FL ports.
The ISLs between the switches should be using a standard based solution. Which means they should use
a normal ISL protocol and not the Cisco specific (although also standardized now, nobody implements it)
EISL encapsulation. We can use the maximum interface bandwidth, which is 4Gbps on our interfaces
between the switches.
Pay attention that you need to use a dedicated rate-mode on certain switches for configuring E -ports.
This has to do with the way that the ASIC is used. On E-ports all B2B credits are assigned to this interface
as its
MDS1(config-if)# int fc1/12
MDS1(config-if)# sw mode e
fc1/12: (error) Auto/E mode is not allowed in shared rate-mode
On E-ports all B2B credits located on the ASIC are assigned to this interface as its required for high
bandwidth and longer links.
217
Finally, all interfaces on the MDS switches are by default set to shutdown (unless specifically
configured).
MDS1
MDS1(config)# int fc1/5
MDS1(config-if)# sw mode fl
MDS1(config-if)# sw speed 2000
MDS1(config-if)# no shut
MDS1(config-if)# sw trunk mode
off
off
off
off
Keep in mind that by default the EISL protocol is enabled on E ports, so ports between MDS switches
automatically become TE ports, when nothing is changed.
MDS1(config-if)# do sh int fc1/12
fc1/12 is down (Link failure or not-connected)
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:0c:00:0d:ec:6f:4f:40
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port vsan is 1
Receive data field Size is 2112
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 frames input, 172 bytes
0 discards, 0 errors
0 CRC, 0 unknown class
0 too long, 0 too short
1 frames output, 172 bytes
0 input OLS, 0 LRR, 0 NOS, 0 loop inits
0 output OLS, 0 LRR, 0 NOS, 0 loop inits
218
MDS1(config-if)#
We are now using standard based ISLs and we configured a few JBODs to the MDS switches. If
everything is working correctly. We should see FLOGIs being done to the switch:
MDS1(config-if)# do sh flogi database
-------------------------------------------------------------------------INTERFACE VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------fc1/5
1
0x1b02e8 c6:80:00:11:0d:07:fa:ce
c5:9a:00:06:2b:01:00:07
fc1/5
1
0x1b02ef c6:80:00:11:0d:01:01:07
c5:9a:00:06:2b:01:00:07
Total number of flogi = 2.
MDS1(config-if)#
And its working! We see disks from the JBOD logging in to the MDS switch successfully!
Now we need to configure the second switch. The task here is that the JBOD interfaces should use
automatic selection of speeds, but still should comply with the previous task of 200MBps transfer rates.
This is done by specifying a maximum speed of the automatic selection.
MDS2
MDS2(config-if)# sw speed auto
MDS2(config-if)# sw speed auto
max 2000
max 2000
off
off
off
219
MDS2(config-if)# sw trunk mode off

Next up is that we need to give descriptive names to the ports in the fabric, but we cant use the
command that we are used to use. The MDS switches know besides the description field another
field that can be used for describing the use of the interface. This feature is known as the port
owner. Same as for the description this field can exist of 80 characters describing the owner or the
function of the switch port.
MDS1
MDS1(config-if)# sw owner JBOD1A
MDS1(config-if)# sw owner JBOD2A
MDS1(config-if)# sw owner MDS2 port
1
2
3
4
MDS2
MDS2(config-if)# sw owner JBOD1B
MDS2(config-if)# sw owner JBOD2B
1
2
3
4
One of the ports on MDS1 should be easily located. This is done using a blinking switchport LED known
as the beacon. This is a single command which enables you to physically find interfaces in high port
density switches.
MDS1
MDS1(config-if)# sw beacon
The next feature is a feature used for long distance links. Those links are typically using lots of
interconnects and it could occur that errors are risen during the transport of a frame across this fiber.
The MDS switches use certain thresholds for these errors and after a threshold is reached the interface
is put in err-disabled mode because of this.
220
The thresholds are reached when 15 error bursts occur in a 5 minute period. The errors might occur
because of one of the following reasons:
Bad fiber
Bad SFP
Interface is specified to operate at 1 Gbps, but is used at 2 Gbps
Interface is specified to operate at 2 Gbps, but is used at 4 Gbps
Short haul cable is used for long haul or long haul cable is used for short haul
Momentary sync loss
Loose cable connection at one or both ends
Improper SFP connection at one or both ends
We can disable this if we know the link is fine and we want to overrule the thresholds. As there
is no MDS specified in this task we will configure both.
MDS1
MDS1(config-if)# sw ignore bit-errors
MDS2
MDS2(config-if)# sw ignore bit-errors
Next we need to ensure that all interfaces are shutdown without any configuration. This is where we are
configuring defaults.
MDS1
MDS1(config)# system default ?
switchport Configure default values for switchport attributes
zone
Configure default values for zone
MDS1(config)# system default switchport ?
mode
Enter the port mode
shutdown Disable/enable switchports by default
trunk
Configure trunking parameters as a default
MDS1(config)# system default switchport shutdown
MDS1(config)#
MDS2
MDS2(config)# system default switchport shutdown
MDS2(config)#
221
The next task is regarding giving specific devices within the Fibre Channel an easily remembered name
or alias. This alias should be known throughout the fabric. We can do this using 2 ways. One of
them is by using a standard based solution called FCalias. This is a method of giving names to
certain characteristics of a connected host (WWPN, Interface, FCID, etc.). The only problem is that
FCalias is dependent on the VSAN. You cannot create a VSAN overlapping FCalias. Besides that its
not possible to distribute the FCalias information throughout the Fibre Channel fabric.
The solution we are looking for here is a Cisco proprietary solution called device-alias. These
device-aliases can be distributed throughout the FC Fabric by using the Cisco Fabric Services (CFS). You
can create a device-alias using a PWWN of the device. By default when you the use this device-alias
in any kind of configuration, for example Zoning, it will automatically convert back to the PWWN of the
device.
What you really want is that the device-alias remains the connection between the zone and the
host, so that when changes are done to the device-alias these are leveraged back into the zoning
database automatically. This is done using so-called Enhanced device-alias. Then the devicealias name is kept in the zoning configuration, just like our task asks for.
Do not forget to enable the distribution of device-aliases, because this is not enabled by default.
You can use the FLOGI database for filling the device-alias database.
MDS1
MDS1(config)# device-alias distribute
MDS1(config)# device-alias mode enhanced
MDS1(config)# device-alias database
MDS1(config-device-alias-db)# device-alias
c6:83:00:11:0d:18:fa:ce
c6:83:00:11:0d:18:fb:ce
c6:83:00:11:0d:18:fc:ce
c6:83:00:11:0d:18:fd:ce
21:00:00:11:0d:18:fe:ce
21:00:00:11:0d:18:ff:ce
21:00:00:11:0d:18:a0:ce
21:00:00:11:0d:18:a1:ce
name J1D1 pwwn

name J1D2 pwwn
name J1D3 pwwn
name J1D4 pwwn
name J2D1 pwwn
name J2D2 pwwn
name J2D3 pwwn
name J2D4 pwwn
commit
Now we need to verify if we see all the aliases on the other MDS as well, without configuring it. Be
aware that your ISLs are up before this works!
222
MDS1
MDS1(config)# do sh device-alias database
device-alias name J1D1 pwwn c6:83:00:11:0d:18:fa:ce
device-alias name J1D2 pwwn c6:83:00:11:0d:18:fb:ce
device-alias name J1D3 pwwn c6:83:00:11:0d:18:fc:ce
device-alias name J1D4 pwwn c6:83:00:11:0d:18:fd:ce
device-alias name J2D1 pwwn 21:00:00:11:0d:18:fe:ce
device-alias name J2D2 pwwn 21:00:00:11:0d:18:ff:ce
device-alias name J2D3 pwwn 21:00:00:11:0d:18:a0:ce
Total number of entries = 8
MDS1(config)#
MDS2
MDS2(config)# do sh device-alias status
Fabric Distribution: Enabled
Database:- Device Aliases 8 Mode: Enhanced
Checksum: 0x617152b263ae5b4ebb547a6fb633b7fb
MDS2(config)#
MDS2(config)# do sh device-alias database
MDS2(config)#
B2B credit buffers are assigned to an interface. The buffers are used for the fact of the no-drop policy
that is used in FC fabrics. When a packet is sent, it costs 1 credit on the sending side. This credit is
returned to the pool when an acknowledgement is received from the other side. When the switch runs
out of buffer credits, it will not send any new packets on the interface until it receives
acknowledgements.
The maximum number of credits is very much dependent on the switch type and the line card that is
used for the interface. There are many configuration options for sharing credits in an ASIC between
different ports.
The recovery that is asked for has to do with the fact that sometimes the acknowledgements might get
lost on longer interfaces due to bit errors. This means that in time, the amount of buffer credits gets
less on the interface, while this is not right, as the receiving switch has space enough. This is called the
recovery of credits to get them back periodically. This process is disabled by default.
223
Be aware that the credits that you configure are transmitted to the other end of the interface. The
buffer you configure is the receive buffer, this receive buffer is then communicated so the other side
knows how much buffer space it can fill up by sending frames.
We can verify this by checking the hardware options of the interface and in which port-group its
located:
MDS1# show port-resources module 1
Module 1
Available dedicated buffers are 3765
Port-Group 1
Total bandwidth is 12.8 Gbps
Total shared bandwidth is 8.8 Gbps
Allocated dedicated bandwidth is 4.0 Gbps
-------------------------------------------------------------------Interfaces in the Port-Group
B2B Credit Bandwidth Rate Mode
Buffers
(Gbps)
-------------------------------------------------------------------fc1/1
16
4.0 shared
fc1/2
16
4.0 shared
fc1/3
16
4.0 shared
fc1/4
16
4.0 shared
fc1/5
250
4.0 dedicated
fc1/6
16
4.0 shared
Port-Group 2
MDS1#
Interface fc1/1 is located in port-group one. In this case, the amount of buffer credits is limited to the
module. Some other modules have limitations per port-group.
In this case we have 3765 credits to spare on an interface. The problem is that we can only use up to
250 credits for E-ports. We can however use Extended B2B credit buffers for this use. When
the regular credits run out, we are able to use the extended buffers up to the maximum. Finally we
disable the credit recovery.
You require the Enterprise license to activate the Extended B2B credit buffers. Remind
that you need to enable the feature separately.
MDS1
MDS1(config)# feature fcrxbbcredit extended
MDS1(config-if)# switch fcrx
fcrxbbcredit
fcrxbufsize
MDS1(config-if)# switch fcrxbbcredit ?
<1-500>
Enter receive BB_credit
default
Default receive BB_credit
224
extended
performance-buffers
Configure extended BB_credit for the port

Configure performance buffers for receive BB_credit
MDS1(config-if)# switch fcrxbbcredit extended ?

<256-4095> Enter extended credit receive BB_credit
MDS1(config-if)# switch fcrxbbcredit extended 4095
fc1/5: (error) Receive bbcredit unavailable
We cant configure that much buffers as we just dont have them available. We configure the maximum
that remains after subtracting all other interface credits.
MDS1(config-if)# switch fcrxbbcredit extended 4012
MDS1(config-if)# switchport fcbbscn
As the final step we enable credit recovery, by enabling the BB_SC_N field towards the other side.
Now we verify that we are indeed using that much credits on the interface. As we can see the other end
communicated 250 credits to our interface.
MDS1(config-if)# do sh run int fc1/1
!Command: show running-config interface fc1/1
!Time: Wed Oct 17 16:43:04 2012
version 5.2(6)
interface fc1/1
switchport rate-mode dedicated
switchport fcrxbbcredit extended 4012
switchport mode E
no shutdown
MDS1(config-if)# sh int fc1/1 bbcredit
fc1/1 is trunking
Transmit B2B Credit is 32
Receive B2B Credit is 4012
4012 receive B2B credit remaining
32 transmit B2B credit remaining
32 low priority transmit B2B credit remaining
MDS1(config-if)# do sh port-resources module 1
Module 1
Available dedicated buffers are 0
Port-Group 1
-------------------------------------------------------------------Interfaces in the Port-Group
B2B Credit Bandwidth Rate Mode
Buffers
(Gbps)
-------------------------------------------------------------------fc1/1
4012
4.0 dedicated
fc1/2
16
4.0 shared
fc1/3
16
4.0 shared
225
fc1/4
fc1/5
fc1/6
16
16
16
4.0
4.0
4.0
shared
dedicated
shared
Port-Group 2
MDS1(config-if)#
The Maximum Transmission size is also a configuration item that is communicated to the other
end. Meaning that you do not configure the Maximum Transmission Size on the interface, but
you configure the Maximum Receive Size of the packet, because you (as in the switch) know how
much buffer space you have to receive Fibre Channel frames. The MTU size is by default configured to
the maximum supported in FC environments which is 2112 bytes.
MDS1
MDS1(config-if)# switch fcrxbufsize ?
<256-2112> Enter receive buffer size
MDS1(config-if)# switch fcrxbufsize 2000
As a final task we need to configure State Change Numbers on all JBOD interfaces. This field is
carried in the same BB_SC_N field as the credit recovery is handled. So we need to enable this on all
the JBOD connected interfaces.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
int fc1/5
switch fcbbscn
int fc1/6
switch fcbbscn
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
int fc1/5
switch fcbbscn
int fc1/6
switch fcbbscn
We completed our first Fibre Channel task!
226
Task 2: VSANs
The next task is regarding the virtualization feature that is supported on Ciscos MDS switches. This
feature enables the use of different Fibre Channel SANs on the same physical hardware.
This is done using a tagging mechanism which is very similar to 802.1Q tagging in the Ethernet world.
The technical similarity to a VLAN + a VRF in the Ethernet world. As there is a full Layer 2 separation
between the different Fibre Channel Fabrics. In Ethernet terms we would also say that there is a Layer 3
separation as the routing protocol (FSPF) is also separated.
During this task we will be using the VSANs and the routing protocol FSPF to steer traffic across
different ways through our fabric. Another topic in this task is port-channeling of ISL links.
All settings on the MDS are VSAN dependent. Therefore there is a total separation between controlplanes of the VSANs. This introduces a way that you need to specify the VSAN in each of the show and
configuration commands that you type in.
This also means that Zoning in one VSAN has absolutely no impact on Zoning in other VSANs.
The drawing below illustrates the use of VSANs, by using the trunking between MDS switches, you can
connect devices in any VSAN on any MDS switch and transport the FC frames between switches by using
the EISL tagging (Enhanced ISL) on interfaces.
227
Throughout this task explanation, topics will be explained during the tasks that are discussed.
The first tasks are regarding the creation of VSANs and we should place some interfaces into the VSANS.
Just like VLANs, interfaces are placed inside a VSAN.
MDS1
MDS1(config)# vsan
MDS1(config-vsan)#
MDS1(config-vsan)#
MDS1(config-vsan)#
MDS1(config-vsan)#
MDS1(config-vsan)#
MDS1(config-vsan)#
database
vsan 10 name IPX_VSAN_10
vsan 10 interface fc1/5
MDS2
MDS2(config)# vsan
MDS2(config-vsan)#
MDS2(config-vsan)#
MDS2(config-vsan)#
MDS2(config-vsan)#
MDS2(config-vsan)#
MDS2(config-vsan)#
database
By default interfaces are placed into a VSAN. When you want to support the moving of certain hosts
throughout the fabric, you can use a technology called Dynamic Port VSAN Membership
(DPVM). This technology will place a port into a VSAN when a certain PWWN is seen.
The DPVM technology can also be distributed throughout the fabric, which is enabled by default, so that
its independent of the switch where it comes online and its automatically placed into the correct VSAN.
You can also enable the auto-learn feature which enables the automatic learning of all devices coming
online on interfaces. These are then placed into the DPVM database and the next time they come online
on another interface, the interface is placed into that new VSAN.
The DPVM configuration overrules the interface VSAN configuration, so the DPVM database is leading in
this case.
Keep in mind that the only port where this works on is F-ports.
We start with the 2 tasks that place either a WWPN or a Device-Alias in the database for DPVM.
MDS2
MDS2(config)# feature dpvm
Enable DPVM first on the second switch. If you commit the changes on one switch and enable it later on
another switch, you need to commit again. So pay attention to when you configure things that are
distributed. This feature and the distribution of it, needs to be enabled on all switches first, before
information is actually synchronized.
228
MDS1
MDS1(config)# feature dpvm
MDS1(config)# dpvm distribute
MDS1(config)# dpvm database
MDS1(config-dpvm-db)# pwwn 20:11:00:0a:31:00:aa:de vsan 30
MDS1(config-dpvm-db)# device-alias J1D1 vsan 40
MDS1(config-dpvm-db)# dpvm activate
MDS1(config)# dpvm commit
Now we can see that all information is synchronized between the switches. By using the status
command you can always check the configuration of the CFS services running for this particular feature.
Very useful when troubleshooting problems with the CFS distribution.
MDS2
MDS2(config)# do sh dpvm status
No active DB, auto-learn is off, distribution is enabled,
Duplicated pwwn will be Rejected.
MDS2(config)#
After configuring MDS1, we see that everything is synchronized between the switches
MDS2(config)# do sh dpvm database
pwwn 20:11:00:0a:31:00:aa:de vsan 30
device-alias J1D1 [20:11:00:0a:31:00:aa:df] vsan 40
[Total 2 entries]
MDS2(config)# do sh dpvm status
DB is activated successfully, auto-learn is off, distribution is enabled,
Duplicated pwwn will be Rejected.
MDS2(config)#
The next questions are related to the load balancing that is used within a VSAN to distribute information
across the fabric. This load balancing mechanism is used for both Equal Cost Multi Pathing
(ECMP) when 2 equal costs exist in the FSPF database or when a Port-channel is used with
multiple links bundled as one logical interface.
The load-balancing is configured per VSAN. It can be configured in 2 ways.
Source and Destination FC-ID
This method ensures that flows between a source and destination pair is always using the same
link
Source and Destination FC-ID and the Originator Exchange ID OX-ID
The OX-ID is a value in the Fibre Channel header that is used to identify a traffic flow. Every
exchange, or every action, done between 2 devices gets a new value attached. All these
values are separated across different hashes so this mechanism achieves a much more granular
load balancing.
229
In this task VSAN 10 should use the basic Source and Destination FCID where as VSAN 20
should use exchange based load balancing. This is configured using 2 relatively easy
commands on both switches:
MDS1
MDS1(config)# vsan database
MDS1(config-vsan)# vsan 10 loadbalancing src-dst-id
MDS1(config-vsan)# vsan 20 loadbalancing src-dst-ox-id
MDS2
MDS2(config-vsan)# vsan 10 loadbalancing src-dst-id
MDS2(config-vsan)# vsan 20 loadbalancing src-dst-ox-id
All ISLs should then be made possible to transport all the different VSANs that we created. This means
we now need to enable the trunking on the ISLs to enable this.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
interface fc1/1
switchport trunk
interface fc1/2
switchport trunk
interface fc1/3
switchport trunk
interface fc1/4
switchport trunk
mode on
mode on
mode on
mode on
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
interface fc1/1
switchport trunk
interface fc1/2
switchport trunk
interface fc1/3
switchport trunk
interface fc1/4
switchport trunk
mode on
mode on
mode on
mode on
To verify the working of the trunks, we can issue a similar command to what we use in the Ethernet
world.
MDS1(config)# do sh int trunk vsan
fc1/1 is trunking
Vsan 1 is up (None)
Vsan 10 is up (None)
fc1/2 is trunking
Vsan 1 is up (None)
230
fc1/3 is trunking
Vsan 1 is up (None)
fc1/4 is trunking
Vsan 1 is up (None)
MDS1(config)#
The next questions are regarding the bundling of interfaces in single logical bundles. This is where the
load balancing is used between the different members of port-channels.
Port-channeling on the MDS switches is also a unique feature of the Cisco MDS platform. These
port-channels have equal options as the standard FC interfaces.
The port-channels also support a protocol to negotiate there possibilities, just like LACP in the 802.3ad
protocol for Ether Channels. Almost the same rules are used for including interfaces in a bundle. The
speed, ASIC rate-mode, BB credits, Trunking or Non-Trunking mode and allowed
VSANs need to be equal.
Its possible to channel both E (TE) and F (TF) ports, when the platforms support it. One place where
this is commonly seen is in environments that uses N-Port Virtualization (NPV).
We start by configuring the first port-channel between the 2 MDS switches. This is done in a relatively
equal way to Ethernet port-channels on the Nexus switches.
MDS1
MDS1(config-if)# interface port-channel 101
MDS1(config-if)# switchport mode e
MDS1(config-if)# switchport trunk mode on
MDS1(config-if)# interface fc1/1
MDS1(config-if)# channel-group 101
fc1/1 added to port-channel 101 and disabled
please do the same operation on the switch at the other end of the portchannel,
then do "no shutdown" at both ends to bring it up
231

MDS2
Now we brought up our first port-channel. Keep in mind that you might need to shut and no-shut
certain interfaces as the ports might get suspended when you are configuring the other switch for the
port-channeling.
MDS1(config-if)# do sh port-channel summ
----------------------------------------------------------------------------Interface
Total Ports
Oper Ports
First Oper
Port
----------------------------------------------------------------------------port-channel 101
2
2
fc1/1
MDS1(config-if)#
MDS1(config-if)# do sh int port-channel 101
port-channel 101 is trunking (Not all VSANs UP on the trunk)
Hardware is Fibre Channel
Port WWN is 24:65:00:05:9b:70:5a:40
Port mode is TE
Port vsan is 1
Speed is 8 Gbps
Trunk vsans (admin allowed and active) (1,10,20,30,40)
Trunk vsans (up)
()
Trunk vsans (isolated)
()
Trunk vsans (initializing)
(1,10,20,30,40)
232

Member[1] : fc1/1
Member[2] : fc1/3
Interface last changed at Wed Oct 17 22:07:32 2012
MDS1(config-if)#
Now we also need to make them auto negotiate their bundling capabilities. These capabilities activate a
protocol similar to LACP. Instead of configuring it under the member interfaces, this mode is changed
under the port-channel configuration.
MDS2
MDS2(config-if)# int port-channel 101
MDS2(config-if)# channel ?
mode Set the channel mode for the port-channel interface
MDS2(config-if)# channel mode ?
active Configure ACTIVE port-channel
MDS2(config-if)# channel mode active
MDS2(config-if)#
MDS2
MDS1(config-if)# do sh port-channel database
port-channel 101
Administrative channel mode is on
Operational channel mode is on
Last membership update succeeded
First operational port is fc1/1
2 ports in total, 2 ports up
Ports:
fc1/1
[up]
fc1/3
[up] *
MDS1(config-if)# channel mode active
MDS1(config-if)# do sh port-channel database
port-channel 101
Administrative channel mode is active
Operational channel mode is active
Last membership update succeeded
First operational port is fc1/1
2 ports in total, 2 ports up
Ports:
fc1/1
[up]
fc1/3
[up] *
MDS1(config-if)#
233
The next task is asking about another channel. In the past the MDS switches knew a mechanism which
allowed the auto creation of port-channels. You could enable a single command on a range of interfaces
and the switch then automatically detects which ports are eligible for bundling towards the same other
MDS switch on the other end.
This creates a very easy way to configure link bundles. You could also configure the interface after its
been created with changes to the whole bundle. The number that was used for the channel depends on
the maximum number of supported port-channels on that particular platform, as it uses the last
available number.
However, as of NX-OS 4.1(1b), this feature was removed as it also introduced issues in that you didnt
know where the port-channels were pointing to and you really had to know what was going on the
switches.
We configure the remaining 2 ISLs in another port-channel.
MDS1
MDS2
234
Now what remains is that we need to change our trunks to be using different links. The first 2 questions
are related to the allowed VSAN command, which enables which VSANs are allowed to transfer across
the links.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
interface port-channel 101

sw trunk allowed vsan 10
sw trunk allowed vsan add 20
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#

Now to verify we only see the VSANs that we configured under the interfaces to be trunked to the other
MDS.
MDS2(config-if)# do sh int trunk vsan
port-channel101 is trunking
Vsan 20 is down (Initializing)
MDS2(config-if)#
235
The other 2 questions are related to the steering of traffic across the fibre channel fabric. The focus in
this question is about the FSPF protocol that is working on all FC switches. This protocol is very closely
connected to the OSPF protocol and works basically the same. The easy part is that it works fully
automatically. There is no configuration necessary and will immediately start creating neighbors and
exchange information. The only information that is exchanged in the FSPF protocol is the Domain ID
of the switch in the specific VSAN. The Fibre Channel routing uses those FC Domain IDs to steer
traffic to the correct switch. After it reaches the destination switch it will perform a look-up to find the
egress interface of the traffic where it can send its traffic to. This is all bound to a VSAN basis. Therefore
a VSAN can also be called a Layer 3 VRF in the IP world, as the routing protocol is also fully separated.
We can steer the traffic by changing the FSPF link cost between the switches. FSPF link cost is dependent
on the bandwidth on the link. Therefore also for port-channels the amount of bandwidth of all member
interfaces is counted up to a high bandwidth number.
The following list shows the default FSPF link cost:
1Gbps: 1000
2Gbps: 500
4Gbps: 250
8Gbps: 125
16Gbps (port-channel of 2 x 8gbps links): 62
As we are working with 8Gbps links we have a default cost of 125. Therefore we will lower the cost on
the links we want to make active for a certain VSAN.
We need to make VSAN10 to use port-channel101 and VSAN20 to use port-channel 102 as their primary
links.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
interface
fspf cost
interface
fspf cost
port-channel 101
99 vsan 10
port-channel 127
99 vsan 20
interface
fspf cost
interface
fspf cost
port-channel 101
99 vsan 10
port-channel 127
99 vsan 20
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
236
To verify we need to use a little strange command to find out which interface is actually used for the
traffic. By default the command doesnt show which interface is used. We do see the correct FSPF cost
already being in place.
MDS2# sh fcroute unicast
D:direct
R:remote
Protocol
-------local
fspf
fspf
fspf
fspf
MDS2#
MDS2#
VSAN
---1
10
20
30
40
P:permanent
V:volatile
FC ID/Mask
-------- -------0x290000 0xffffff
0x880000 0xff0000
0x3a0000 0xff0000
0x260000 0xff0000
0x4f0000 0xff0000
A:active
RCtl/Mask
---- ---0x00 0x00
0x00 0x00
0x00 0x00
0x00 0x00
0x00 0x00
N:non-active
# Next
Flags Hops Cost
----- ---- ---D P A 1
1
D P A 1
99
D P A 1
99
D P A 1
250
D P A 1
250
When we look a little deeper, by specifying the destination FC Domain ID, we can see the actual
interface that is used for the next-hop. There we see that we are having a properly configured FSPF
network!
MDS2# sh fcroute unicast host 0x880000 0xff0000 vsan 10
D:direct
R:remote
P:permanent
V:volatile
Protocol VSAN
FC ID/Mask
-------- ----------- -------fspf
10
0x880000 0xff0000
port-channel101 Domain 0x88(136)
MDS2#
MDS2# sh fcroute unicast host 0x3a0000
D:direct
R:remote
P:permanent
RCtl/Mask
---- ---0x00 0x00
N:non-active
# Next
Flags Hops Cost
----- ---- ---D P A 1
99
0xff0000 vsan 20
V:volatile
Protocol VSAN
FC ID/Mask
-------- ----------- -------fspf
20
0x3a0000 0xff0000
port-channel127 Domain 0x3a(58)
MDS2#
A:active
A:active
RCtl/Mask
---- ---0x00 0x00
N:non-active
# Next
Flags Hops Cost
----- ---- ---D P A 1
99
The next question is regarding the possibility of the FC Fabric to deliver packets in-order between hosts.
Some hosts that are connected to the fabric require this, for example HP/UX hosts or FICON hosts
assume that the fabric will deliver packets in-order and will have various issues when traffic is not
delivered in the same order as it was sent.
Therefore you can enable the in-order delivery of packets per VSAN.
MDS1
MDS1(config)# in-order-guarantee vsan 30
237
MDS2
MDS2(config)# in-order-guarantee vsan 30
Its even possible to create static routes in the FC fabric network by using the fcroute command. We
will need this kind of traffic engineering for the next question to send traffic between two
specific disks across another link than the default traffic is going.
In this question you should be aware that its talking about 2 ways of traffic, so the steering needs to be
set back to disk as well (bidirectional traffic flow).
Pay close attention to the FC Domain ID of the switch and the port. These can change when the host is
re-connected or the host is rebooted and the Domain ID is not statically configured. In this question its
not that relevant, but in the lab, keep this in mind. You will want to have static FCIDs configured when
you need to create a configuration like this.
MDS2
MDS2(config)# fcroute 0x8800e8 0xffffff interface port-channel127 domain
136 ?
aggregate Aggregable route
metric
Cost of route
remote
Destination switch remotely connected
vsan
VSAN id
When you specify a destination Domain ID of anything behind the next-hop switch, you should
specify the remote keyword here. As otherwise the switch will always assume that the domain ID
specified is also the next-hop switch. Look up this domain ID using the show fcdomain vsan
command.
MDS1(config-vsan-db)# do sh fcdomain vsan 10
The local switch is the Principal Switch.
Local switch run time information:
State: Stable
Local switch WWN:
20:0a:00:05:9b:70:5a:41
Running fabric name: 20:0a:00:05:9b:70:5a:41
Running priority: 2
Current domain ID: 0x88(136)
Local switch configuration information:
State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled
Contiguous-allocation: Disabled
Configured fabric name: 20:01:00:05:30:00:28:df
Optimize Mode: Disabled
Configured priority: 128
Configured domain ID: 0x00(0) (preferred)
Principal switch run time information:
Running priority: 2
238
MDS1(config-vsan-db)#
MDS2
MDS2(config)# fcroute 0x8800e8 0xffffff interface port-channel127 domain
136 vsan 10
MDS1
MDS1(config)# fcroute 0x7f0000 0xffffff interface port-channel127 domain
127 vsan 10
The final verification is again by using the fcroute show command, so we can check specific
destinations, not only for the Domain ID mask, but also for the host mask.
MDS2(config)# show fcroute unicast host 0x8800e8 0xffffff vsan 10
D:direct
R:remote
P:permanent
V:volatile
Protocol VSAN
FC ID/Mask
-------- ----------- -------static
10
0x8800e8 0xffffff
fspf
10
0x880000 0xff0000
MDS2(config)#
A:active
RCtl/Mask
---- ---0x00 0x00
N:non-active
# Next
Flags Hops Cost
----- ---- ---D P A 1
10
0x00 0x00
D P A
99
In this show output we see that all traffic is sent using the FSPF route across port-channel 101
and that traffic specific for the disk in the JBOD is sent across the other port-channel. Just what the
question stated!
In the Fibre Channel network there is also a root switch for all the Multicast and Broadcast traffic that
is sent throughout the fabric. By default the root switch is also the Principal switch of the fabric. In
case of VSAN 20 we need to change this to be the lowest domain ID switch
This is done using a single command, again VSAN specific.
MDS1
MDS1(config)# mcast root lowest vsan 20
MDS2
MDS2(config)# mcast root lowest vsan 20
To verify we see that by default the Principal switch is chosen as the default multicast root. Now for
VSAN 20 we changed this to be the lowest Domain ID switch.
MDS2(config)# show mcast vsan 10
Multicast root for VSAN 10
Configured root mode : Principal switch
Operational root mode : Principal switch
Root Domain ID : 0x88(136)
239
MDS2(config)# show mcast vsan 20

Multicast root for VSAN 20
Configured root mode : Lowest domain switch
Operational root mode : Lowest domain switch
Root Domain ID : 0x3a(58)
MDS2(config)#
The next task is regarding so-called iSPF calculations. With these incremental changes the SPF
calculations of the FSPF protocol are done only on the changes.
MDS2
MDS2(config)# fspf config vsan 30
MDS2(config-(fspf-config))# ?
min-ls-arrival
Configure Min LS Arrival
min-ls-interval Configure Min LS interval
no
region
Configure the autonomous region for FSPF
spf
Configure parameters related to SPF route computation
end
Go to exec mode
exit
pop
push
where
MDS2(config-(fspf-config))# spf ?
hold-time Configure the hold time between route computations
static
Force static spf computation
MDS2(config-(fspf-config))# spf static
MDS1
MDS1(config)# fspf config vsan 30
MDS1(config-(fspf-config))# spf static
The final question in this task is that we want unused ports not to use the Default VSAN. The
Default VSAN is always VSAN 1. Now unused ports are by default configured into VSAN 1.
This cannot be changes easily, but we can ensure that there is no traffic being sent within VSAN 1. This
is done by making it suspended, but pay close attention! This makes the process difficult as all
interfaces start from VSAN 1 including all ISLs.
To overcome this limitation is to configure the ports in another VSAN just to get them coming online.
Keep this also in mind in later chapters where you are configuring additional links in the FC fabric,
like FCIP tunnels, etc.
MDS1
MDS1(config-vsan-db)# vsan 1 suspend
240
MDS2
MDS2(config-vsan-db)# vsan 1 suspend
Now initially everything looks fine and the trunks are still up and running.
MDS2(config-vsan-db)# do sh int trunk vsan
fc1/5 is trunking
fc1/6 is trunking
However when the links are re-enabled, they will not come back online and you will get into a very tricky
and stressful situation when you are far down the lab!
MDS2(config-if)# do sh int fc1/5
fc1/5 is down (Inactive)
Port WWN is 20:05:54:7f:ee:01:2b:80
Port vsan is 1
This problem is overcome by giving the ISL links another VSAN in the VSAN database so they are made
online within another VSAN, before the trunking protocol is initialized.
MDS1
MDS1(config-vsan-db)# vsan 10 interface port-channel101
MDS1(config-vsan-db)# interface port-channel101
MDS1(config-if)# shut
MDS1(config-if)# interface port-channel127
241
MDS2
After the re-configuration of the ISL links, the port-channel interfaces are coming online just as they
were with the correct trunking protocol and we are up and running again!
MDS1(config-vsan-db)# do sh int fc1/5
fc1/5 is trunking (Not all VSANs UP on the trunk)
Port WWN is 20:05:00:05:9b:70:5a:40
Peer port WWN is 20:05:54:7f:ee:01:2b:80
Port mode is TE
Port vsan is 10
Speed is 4 Gbps
Rate mode is dedicated
B2B State Change Number is 14
Trunk vsans (admin allowed and active) (10,20,30,40)
Trunk vsans (up)
(10,20)
Trunk vsans (isolated)
(30)
Trunk vsans (initializing)
(40)
Again this error is not spotted when you are still working on the switches and they are not flapped. Once
they are flapped or the switch is rebooted the links will remain down unless they are given another
interface in the VSAN database like we just changed.
242
The final question of this task is for using another protocol in our Fibre Channel Fabric.
This protocol is called the IP over Fibre Channel protocol (IPFC), which is very different
from the FCIP protocol!
This protocol is to create IP interfaces across the FC infrastructure. You can use different protocols for
this use, one of them is management of the FC fabric network or for using CFS over IP services.
The configuration is very similar to creating Switched Virtual Interfaces (SVIs) in the
Ethernet world. Dont forget to name the VSAN as this is a requirement from the beginning of this task.
MDS1
MDS1(config)# vsan data
MDS1(config-vsan-db)# vsan 50 name IPX_VSAN_50
MDS1(config-if)# sw trunk allowed vsan add 50
MDS1(config-if)# int port-channel127
MDS1(config-if)# interface vsan 50
MDS1(config-if)# ip add 198.18.50.1 255.255.255.0
MDS2
MDS2(config)# vsan data
MDS2(config-vsan-db)# vsan 50 name IPX_VSAN_50
MDS2(config-if)# int port-channel127
MDS2(config-if)# interface vsan 50
After the configuration, we can see that the new VSAN is allowed on the Trunk interfaces and we can
ping the other MDS switch! This means that the IPFC connection is working.
MDS2(config-if)# do sh int trunk vsan
MDS2(config-if)# do ping 198.18.50.1
PING 198.18.50.1 (198.18.50.1) 56(84)
64 bytes from 198.18.50.1: icmp_seq=1
bytes of data.
ttl=64 time=5.16 ms
ttl=64 time=2.04 ms
ttl=64 time=2.03 ms
243

--- 198.18.50.1 ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 5050ms
rtt min/avg/max/mdev = 2.014/2.552/5.162/1.167 ms
MDS2(config-if)#
Now task 2 is finished and we continue with the next task about Zoning!
Task 3: Zoning
This third task will be about configuring a typical Fibre Channel feature. By default nobody in the FC
network can communicate to each other! This is a very important topic about Fibre Channel. You can
compare Zoning to Access Lists in the IP world. By default there is a deny rule on all the
interfaces.
The zoning needs to be configured per VSAN. All devices that are configured under a zone are able to
communicate to each other. Devices between zones cant talk to each other. Zones are combined
under a Zoneset. This zoneset is activated on a specific VSAN and spread throughout the FC Fabric.
When devices are not specified within a zone, they are part of the Default Zone. This Default
Zone will not process any traffic unless specifically configured in the CLI. Devices can be configured
under multiple Zones at the same time. For example a certain server needs access to various JBODs,
but the JBODs can not communicate to each other. This would result in a number of zones each
specifying the same server and 1 JBOD.
The standard zoning was not sufficient for larger environments. This is why Enhanced Zoning was
created. With Enhanced Zoning it becomes possible to perform zoning configuration from a
single switch at the same time. With basic zoning, administrators can override each others changes
easily by just applying a different zoneset to the VSAN. With Enhanced zoning, all information is
distributed within the fabric and the fabric is locked for changes when one administrator starts
configuring zoning.
We will start by configuring relatively easy zoning and will build up to more complex zoning including
advanced features.
One important thing to remember with zoning is that you should never forget to change the zoning
when this is required. When we will be configuring the UCS system later on, we might require some
additional zoning to be done, while this might not be specifically mentioned in the lab task, however
you can not access your storage without any configuration about the zoning.
The first question in this task is by creating a task by using the device-alias of the devices. Since we
already enabled Enhanced Device-Aliases, we will keep seeing these names in the zoning
configurations.
244
Pick easy to remember names for your zones. You could create zone names with the devices in it, but
usually whats best to just use a simple number in the names to make it clear. You can have overlapping
zone names between VSANs, as they are never created as VSAN independent.
MDS1
MDS1(config)# zone name ZONE1 vsan 10
MDS1(config-zone)# member device-alias J1D2
MDS1(config-zone)# zoneset name ZONESET1 vsan 10
MDS1(config-zoneset)# member ZONE1
MDS1(config-zoneset)# zoneset activate name ZONESET1 vsan 10
Zoneset activation initiated. check zone status
MDS1(config-zone)#
MDS2
MDS2(config)# do sh zoneset active vsan 10
zoneset name ZONESET1 vsan 10
zone name ZONE1 vsan 10
device-alias J1D3
device-alias J1D4
device-alias J1D2
MDS2(config)#
On MDS2 the only thing we need to do is verify, as the zones are distributed throughout the fabric. Pay
attention that only the active zones are distributed! MDS2 does not have the configuration for the
zones in its running-configuration now!
The next task requires you to use something different than de device-aliases. Its possible to
create zones in many different ways, using various options to include devices.
MDS1
MDS1(config-zone)# member ?
device-alias
Add device-alias member to zone
domain-id
Add member based on domain-id,port-number
fcalias
Add fcalias to zone
fcid
Add FCID member to zone
fwwn
Add Fabric Port WWN member to zone
interface
Add member based on interface
pwwn
Add Port WWN member to zone
symbolic-nodename Add Symbolic Node Name member to zone
MDS1(config-zone)# member pwwn 21:00:00:11:0d:18:fe:ce
MDS1(config-zone)# member pwwn 21:00:00:11:0d:18:a1:ce
MDS1(config)#
245
Remember to always activate the zoneset. Without activation the configuration is not working
and devices can still not communicate to each other!
Now when we verify we see that MDS2 can automatically map the configured PWWNs to the devicealias that is also configured for this JBOD disk.
device-alias J1D3
device-alias J1D4
device-alias J1D2
pwwn 21:00:00:11:0d:18:fe:ce [J1D5]
pwwn 21:00:00:11:0d:18:a1:ce [J1D6]
MDS2(config)#
The next task is to add an interface of another MDS switch to the zoning configuration. This can be
done by using 2 factors. Either the Domain ID or the SWWN of the switch. The SWWN is a given fact of
the switch that is not possible to change, so this would be the preferred way to configure this type of
zoning.
MDS2
MDS2(config)# sh wwn switch
Switch WWN is 20:00:00:05:9b:73:57:40
We first do a lookup of the SWWN of the switch before we can configure the new zone.
MDS1
MDS1(config-zone)# member interface ?
fc
Fiber Channel interface
san-port-channel SAN Port Channel interface
MDS1(config-zone)# member interface fc1/6 ?
<CR>
,
Multi range separator
Range separator
domain-id Specify domain-id
swwn
Specify switch-wwn for the interface
MDS1(config-zone)# member interface fc1/6 swwn 20:00:00:05:9b:73:57:40
MDS1(config)#
Then we verify the successful addition of the interface to the zoning.
246

device-alias J1D3
device-alias J1D4
device-alias J1D2
pwwn 21:00:00:11:0d:18:fe:ce [J2D1]
pwwn 21:00:00:11:0d:18:a1:ce [J2D4]
interface fc1/6 swwn 20:00:00:05:9b:73:57:40
MDS2(config)#
Next question is regarding the frames that are sent to the Fibre Channel Broadcast address
(0xFFFFFF). By default broadcast frames arrive to all hosts that are configured in zones. You can
limit this by using broadcast zoning. This broadcast zoning is enabled by using the
broadcast keyword in the zones. When this keyword is configured in one of the zones in the VSAN,
the broadcasting is immediately limited to only the zones that have the configuration applied.
This would solve our question! Do not forget to enable broadcast zoning first.
MDS1
MDS1(config-zone)#
MDS1(config)# zone
MDS1(config-zone)#
MDS1(config-zone)#
MDS1(config)#
zone broadcast enable vsan 10

name ZONE2 vsan 10
attribute broadcast
zoneset activate name ZONESET1 vsan 10
The next question is required to activate your zoning. Best way to do it, would be to activate the
zoneset every time you perform a change. Its difficult to verify what has been enabled and what
hasnt. By just activating it after each change, you will never forget to enable zoning. Again you will
loose points on the lab when this hasnt been done.
The next tasks will let you add another zone to a zoneset, but not change the active zoneset as
well. We still want all of the zonesets and zones to be visible on all switches, which is by default not
done. The process that performs these changes is known as Full Zoneset Distribution.
We can do this in 2 ways, every time a zone distribution is done, this can be performed. In that case the
configuration should be done in the configuration mode, or you can do it only one time, when the
command is issued in the operational mode. To be sure we will do it via the configuration this
time, as the question did not state if it needs to be done only one time or multiple times.
As we can not change any configuration on MDS1 we first need to be sure to have the configuration in
our running-configuration on MDS2. This requires a copy action from the operational CLI first,
247
before starting to configure zones. Otherwise all configuration on the other MDS switch is overwritten
with the new configuration.
MDS2
MDS2# zone copy active-zoneset full-zoneset vsan 10
WARNING: This command may overwrite common zones in the full zoneset. Do
you want to continue? (y/n) [n] y
MDS2# sh run zone vsan 10
!Command: show running-config zone vsan 10
!Time: Thu Oct 18 09:06:25 2012
version 5.0(3)N1(1a)
!Full Zone Database Section for vsan 10
member device-alias J1D3
member pwwn 21:00:00:11:0d:18:fe:ce
member pwwn 21:00:00:11:0d:18:a1:ce
member interface fc1/6 swwn 20:00:00:05:9b:73:57:40
member ZONE1
member ZONE2
member ZONE3
MDS2#
Now all zones are configured in the running-configuration of MDS2. Here we can change the
zoning as required for the question.
MDS2
MDS2(config-zone)# member fcid ?
<0x0-0xffffff> Enter FCID
MDS2(config-zone)# member fcid 0x8800aa
MDS2(config-zone)# member fcid 0x8800bb
MDS2(config-zone)# member fcid 0x8800cc
MDS2(config-zone)# member fcid 0x8800dd
MDS2(config-zone)# member fcid 0x8800ee
MDS2(config-zone)# member fcid 0x8800ff
248
MDS2(config-zoneset)# exit
MDS2(config)# zoneset distribute full vsan 10
MDS2(config)# zoneset activate name ZONESET1 vsan 10
MDS2(config)#
MDS1
MDS1(config)# zoneset distribute full vsan 10
Now when we applied the zoneset, we can check on MDS1 and see that indeed, the non-active
zoneset is transferred as well as is the new zone, which is not active in the active zoneset! We
do need to activate the full zoneset distribution on MDS1 as well, as this setting is not
distributed with the zones.
MDS1# sh zoneset active vsan 10
device-alias J1D3
device-alias J1D4
device-alias J1D2
pwwn 21:00:00:11:0d:18:fe:ce [J2D1]
pwwn 21:00:00:11:0d:18:a1:ce [J2D4]
interface fc1/6 swwn 20:00:00:05:9b:73:57:40
MDS1# sh run zone vsan 10
!Time: Thu Oct 18 08:59:58 2012
member pwwn 21:00:00:11:0d:18:fe:ce
member pwwn 21:00:00:11:0d:18:a1:ce
member interface fc1/6 swwn 20:00:00:05:9b:73:57:40
zone name ZONE4
member fcid
member fcid
member fcid
member fcid
member fcid
member fcid
vsan 10
0x8800aa
0x8800bb
0x8800cc
0x8800dd
0x8800ee
0x8800ff
249

member ZONE1
member ZONE2
member ZONE3
member ZONE1
member ZONE2
member ZONE4
MDS1#
And we completed all zoning for VSAN 10!

For zoning the next VSAN, we will require to use a new type of zoning. Like previously mentioned a
new way of zoning has been introduced in a newer version of the Fibre Channel standards. This type
of zoning is commonly referred to as Enhanced Zoning. The standards that describe this are
FC-GS-4 and FC-SW-3. This big advantage of enhanced zoning is that all changes to the zones
are immediately distributed throughout the fabric. You do still need to perform a manual
activation of the zoning after the configuration. A commit of all changes is not enough to make
them active! We also need to use inline zone creation. This is a way that you can configure
zones right out of the zoneset configuration prompt, making it very easy to use and to configure
multiple zones at once.
We start the configuration by making the first zone, which is also asking for some read-only zoning.
Instead of using the attributes right in the zone, Enhanced Zoning requires to use so-called
attribute-groups that can be applied to zones.
MDS1
MDS1(config)# zone mode enhanced vsan 20
WARNING: This command would distribute the zoning database of this switch
throughout the fabric. Do you want to continue? (y/n) [n] y
Set zoning mode command initiated. Check zone status
MDS1(config)#
After the enhanced zoning has been configured, its seen on all other switches. This change is
immediately distributed.
MDS2(config)# sh zone status vsan 20
VSAN: 20 default-zone: deny distribute: active only Interop: default
mode: enhanced merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:
qos: none broadcast: disabled ronly: unsupported
Full Zoning Database :
DB size: 44 bytes
Zonesets:0 Zones:0 Aliases: 0 Attribute-groups: 1
250
Active Zoning Database :

Database Not Available
Status: Set zoning mode success at 10:03:41 UTC Oct 18 2012
MDS2(config)#
Now the zones can be created inline. We are configuring the first zone.
MDS1
MDS1(config)# zone-attribute-group name RO vsan 20
MDS1(config-attribute-group)# read-only
MDS1(config-attribute-group)# exit
MDS1(config)# zoneset name ZONESET1 vsan 20
Enhanced zone session has been created. Please 'commit' the changes when
done.
MDS1(config-zoneset)# zone ?
name Zone name
MDS1(config-zoneset)# zone name ZONE1
MDS1(config-zoneset-zone)# member device-alias J2D1
MDS1(config-zoneset-zone)# attribute-group RO
MDS1(config-zoneset-zone)# exit
MDS1(config)# zone commit vsan 20
Commit operation initiated. Check zone status
Now we verify that everything is applied correctly

MDS1(config)# sh run zone vsan 20
!Time: Thu Oct 18 09:59:28 2012
zone mode enhanced vsan 20
!Active Zone Database Section for vsan 20
member ZONE1
do clear zone database vsan 20
zone-attribute-group name RO vsan 20
read-only
attribute-group RO
251

member ZONE1
zone commit vsan 20
MDS1(config)#
The next zoning is using a feature which is dedicated to Cisco MDS switches due to the fact that it
requires Trunk E ports between switches to transport a specific field.
Within Cisco MDS switches its possible to use Quality of Service features within zones. By using
these features its possible to give certain hosts or JBODs a higher priority over other traffic. In case of
congestion on the TE-ports between the switches, the traffic marked with a high priority gets served
first, after that the medium priority traffic is served and finally the low priority traffic. The QoS settings
are applied per zone. The marking of the traffic is carried in a special field in the EISL header.
We will configure the next zone with the QoS settings and the FWWNs of the disks instead of their
device-alias or PWWN. You can easily find the FWWN with the show fcns database or show
flogi database on the switch where the JBODs are connected.
Dont forget to use inline zone creation as this was a requirement for the whole of VSAN 20.
MDS1
MDS1(config)# zone-attribute-group name HIGH vsan 20
done.
MDS1(config-attribute-group)# qos ?
priority Priority to be used for frames matching zone
MDS1(config-attribute-group)# qos prio ?
high
Frames matching zone get high priority
low
Frames matching zone get low priority
medium Frames matching zone get medium priority
MDS1(config-attribute-group)# qos prio high
MDS1(config-attribute-group)# zoneset name ZONESET1 vsan 20
MDS1(config-zoneset-zone)# attribute-group HIGH
MDS1(config-zoneset-zone)# member fwwn 22:00:00:11:0d:18:a1:ce
MDS1(config-zoneset-zone)# member fwwn 22:00:00:11:0d:18:a1:de
MDS1(config-zoneset-zone)# zoneset activate name ZONESET1 vsan 20
Now to verify the zones have been applied correctly:

MDS1(config)# sh run zone vsan 20
252
!Time: Thu Oct 18 10:17:46 2012

zone mode enhanced vsan 20
!Active Zone Database Section for vsan 20
member fwwn 22:00:00:11:0d:18:a1:ce
member fwwn 22:00:00:11:0d:18:a1:de
member ZONE1
member ZONE2
do clear zone database vsan 20
zone-attribute-group name RO vsan 20
read-only
zone-attribute-group name HIGH vsan 20
qos priority high
attribute-group RO
attribute-group HIGH
member fwwn 22:00:00:11:0d:18:a1:ce
member fwwn 22:00:00:11:0d:18:a1:de
member ZONE1
member ZONE2
zone commit vsan 20
The next question asks that when devices in VSAN 20 are not configured for zoning, they should still
be able to read data from each other. The only way we can solve this is by allowing traffic between the
hosts that are in the default zone. Besides that they are then immediately able to write data to
each other as well. To stop this, we can change the attributes of the default zone so we can apply
the read-only zoning option as well.
MDS1
MDS1(config)# zone default-zone permit vsan 20
done.
MDS1(config)# zone default-zone vsan 20
253
MDS1(config-default-zone)# read-only
The next topic is regarding a special feature called LUN zoning. This means that when you specify
the LUN ID which needs to be reached behind the zoning, only this LUN is reachable. In this task you
are asked to configure this.
Remember for the LUN numbers, that they are in hexadecimal notation!
MDS1
MDS1(config)# zoneset name ZONESET1 vsan 20
done.
MDS1(config-zoneset-zone)# member device-alias J2D5 lun 0x13
MDS1(config-zoneset-zone)# member device-alias J1D6 lun 0x74
MDS1(config-zoneset-zone)# zoneset activate name ZONESET1 vsan 20
MDS1(config)#
Finally we make the zoning active and we are done!
Task 4: FC Domain
The next task is regarding a technology called FC Domain. The FC Domain process is used within the
FC Fabric to ensure unique IDs are assigned to switches. This Domain ID is then used for
routing the Fibre Channel traffic to other switches. Therefore the Domain ID is a crucial part of the FC
Fabric configuration.
Within a fabric there is a switch chosen to be the Principal switch. This principal switch is
then responsible for handing out the Domain IDs to other switches in the fabric, when they are set to
automatically configuring the Domain ID. The Principal switch is selected based on a
configured priority value and as a tie-breaker the lowest SWWN is used.
The following steps occur when a new FC Fabric is converging:
Principal Switch Election
By using priority values or lowest SWWN a Principal switch of the entire Fibre Channel Fabric
(per VSAN) is chosen.
Domain ID distribution
254
The principal switch ensures that there are only unique Domain IDs in the fabric. Switches can
statically configure their own Domain ID, but they always need to ask the Principal switch if
this is still a unique value in the fabric. When there are duplicate Domain IDs the network will not
work!
When switches are configured to automatically get a Domain ID assigned, the Principal switch
will do this.
FC ID allocation
When Domain IDs are allocated to the switches. The switches can start by assigning FCID blocks
(areas) to the connected devices. They base this on their allocated domain ID. The first byte of the
FCID is the Switch Domain ID. The second byte is the Area code and the last byte is the host
address
Fabric reconfiguration
A domain restart is crucial when new FC Domain IDs are selected for the FC Fabric. This restart is
disruptive for the traffic. All traffic is stopped until the FC Domain and FSPF protocol are fully
converged.
The first tasks are all regarding VSAN 10. MDS1 should be the principal switch and have a static
ID configured. The second switch should prefer a Domain ID.
Pay attention to Decimal and Hexadecimal numbers in these tasks. It depends on the show
command if the Domain ID is presented in decimal or in hexadecimal notation.
By default the FC Domain priority is set to 128. When a principal switch is elected, it changes its
running priority to 2. This is to ensure that it remains the principal switch, even when a
new switch comes into the fabric with a lower SWWN. When we are configuring priority values, we
only need to go under the configured priority of the switches as those are the priorities that
are used in times of convergence of the fabric in the VSAN.
MDS2
MDS2(config)# fcdomain domain ?
<0-239>
Configure the domain id in decimal, 0 not allowed for static
dom
<0x0-0xef> Configure the domain id in hexadecimal
MDS2(config)# fcdomain domain 0x34 preferred vsan 10
MDS1
MDS1(config)# fcdomain priority 99 vsan 10
MDS1(config)# fcdomain domain 34 static vsan 10
MDS1(config)# fcdomain restart disruptive vsan 10
MDS1(config)#
255
We require a Disruptive restart before we will be seeing any results of our configuration
changes.
MDS1
MDS1(config)# sh fcdomain vsan 10
State: Stable
Local switch WWN:
20:0a:00:05:9b:73:1b:c1
Running fabric name: 20:0a:00:05:9b:73:1b:c1
Running priority: 2
State: Enabled
Configured domain ID: 0x22(34) (static)
Running priority: 2
Interface
-----------------port-channel101
port-channel127
-----------------MDS1(config)#
Role
------------Downstream
Downstream
-------------
RCF-reject
-----------Disabled
Disabled
------------
MDS2
The local switch is a Subordinated Switch.
State: Stable
Local switch WWN:
20:0a:00:05:9b:73:57:41
Running fabric name: 20:0a:00:05:9b:73:1b:c1
Running priority: 128
State: Enabled
Configured domain ID: 0x34(52) (preferred)
256

Running priority: 2
Interface
-----------------port-channel101
port-channel127
-----------------MDS2(config)#
Role
------------Upstream
Upstream
-------------
RCF-reject
-----------Disabled
Disabled
------------
We can see that MDS1 is the principal switch of the fabric with the correct priority configured. The
Domain IDs also match with what the question described. Its also shown that the Domain IDs can
be configured in either decimal or hexadecimal format, like they are shown in the show command
as well.
Next we need to ensure that Domain IDs are handed out in a sequential order. By default
Domain IDs are handed out in any order. This change is only necessary on the Principal switch as this
switch is handing out the Domain IDs.
MDS1
MDS1(config)# fcdomain contiguous-allocation vsan 10
MDS1(config)# fcdomain disruptive restart vsan 10
The disruptive restart is a very tough restart. All traffic in the VSAN in the Fabric is interrupted
when this occurs. Therefore there has been made a security mechanism to prevent this from happening
from each switch in the network. This feature enables that only a few switches in the network can
trigger the disruptive restart.
When you configure different Domain IDs in your network a disruptive restart is necessary
to make them active in the fabric. This is because the Domain IDs are used for allocating FCIDs to
the connected hosts. When the Domain ID changes, these have to change as well.
Therefore we limit the disruptive restarts on MDS1. This means that MDS2 can never issue a
disruptive restart for the whole FC Fabric in VSAN 10.
MDS1
MDS1(config)# fcdomain rcf-reject vsan 10
MDS1(config)# fcdomain disruptive restart vsan 10
The next task is regarding a fixed allocation of a certain device connected to MDS1. JBOD1 is connected
to MDS1 in VSAN 10, where this can get assigned static FCIDs.
The IDs that need to be assigned are in the range of the Domain ID that is running on the switch. The
only way to statically configure the FCIDs is when they match with the running Domain ID of
course.
257
We will configure the static allocation of the FCID:

MDS1
MDS1(config)# fcdomain fcid database
MDS1(config-fcid-db)# vsan 10 wwn c6:83:00:11:0d:18:fa:ce fcid 0x222200
area
Now by specifying the WWN of J1D1 and the FCID that we want we can specify the area keyword
behind it. This means that it will assign this entire area to that particular WWN and allocate all FCIDs
just for this disk.
When this disk will perform a new FLOGI to the switch, it will get this FCID assigned!
The next question will assign Domain IDs to the switches in VSAN 20. Again pay attention to the
hexadecimal notation of the allowed FC Domain IDs that can be handed out by the Principal
Switch. In this case MDS2 should be the Principal switch. MDS1 should be requesting a
domain ID.
MDS1
MDS1(config)# fcdomain domain 214 preferred vsan 20
MDS2
MDS2(config)# fcdomain priority 99 vsan 20
MDS2(config)# fcdomain allowed 176-206 vsan 20
Error: attempt to manually set an allowed domain list that would not
include all the currently assigned and locally configured domains or that
would conflict with existing allowed domain lists.
Error: the VSAN is not stable. Please try again after it stabilizes.
Error: attempt to manually set an allowed domain list that would not
include all the currently assigned and locally configured domains or that
would conflict with existing allowed domain lists.
What happens here is that the allowed list cant be set in the VSAN unless all Domain IDs are
currently compliant with the configuration. This means that we need to change the Domain ID of
MDS1 temporarily because this introduces the issue.
MDS1
MDS1(config)# fcdomain domain 214 preferred vsan 20
Once we change the Domain ID of MDS1 back to something that falls within the range that we want
to configure. After the allowed list is configured a final restart of the VSAN is done.
258
MDS2
Now we can verify if everything works like we expect it to work.

State: Stable
Local switch WWN:
20:14:00:05:9b:73:57:41
Running fabric name: 20:14:00:05:9b:73:57:41
Running priority: 2
Current domain ID: 0xb0(176)
State: Enabled
Configured domain ID: 0xb0(176) (static)
Running priority: 2
Interface
-----------------port-channel101
port-channel127
-----------------MDS2(config)#
Role
------------Downstream
Downstream
-------------
RCF-reject
-----------Disabled
Disabled
------------
MDS2 looks fine. We see that it is elected as the Principal Switch of the VSAN and that it is
running a Domain ID that falls within the range.
Now how is MDS 1 working, as we have configured a preferred Domain ID which is outside of the
range that is allowed.
The local switch is a Subordinated Switch.
State: Stable
Local switch WWN:
20:14:00:05:9b:73:1b:c1
Running fabric name: 20:14:00:05:9b:73:57:41
Running priority: 128
Current domain ID: 0xce(206)
State: Enabled
259

Configured domain ID: 0xd6(214) (preferred)
Running priority: 2
Interface
-----------------port-channel101
port-channel127
-----------------MDS1(config)#
Role
------------Upstream
Upstream
-------------
RCF-reject
-----------Disabled
Disabled
------------
The switch is running another Domain ID as the preferred one was not allowed. Therefore the
Principal switch allocated another ID to this switch, which it is using now.
The final question is related to a feature called fast-restart, which has been recently available on
the MDS switches. We need to turn this feature on in VSAN 30. This is one of those features that is
easily looked up in the Documentation on Cisco.com.
MDS1
MDS1(config)# fcdomain optimize fast-restart vsan 30
MDS1(config)#
MDS2
MDS2(config)# fcdomain optimize fast-restart vsan 30
MDS2(config)#
Now task 4 is finished and we have a good working FC Domain configuration.
Task 5: Fibre Channel Security Features

This task will focus on the security features that are available in the Fibre Channel network. This task
focuses on 3 major security features:
Port Security
Port-security ensures that there are no rogue hosts getting connected to the Fibre
Channel Fabric. The feature checks if the device connected has an equal PWWN, FWWN or SWWN
otherwise it is rejected.
260
Port Security can also protect that devices are not connected on different ports. So
devices are only allowed to come online on certain interfaces. This can be one or more.
Fabric Binding
Fabric Binding is a technology used to protect the FC switches in a Fabric. You will need to
have the SWWNs configured of all switches before it may come online in the fabric. You can
choose to even protect the Domain IDs configured on these SWWNs
FC-SP / DH-CHAP
FC-SP is a protocol which is also called DH-CHAP. Configuration commands all start with
DHCHAP. This feature ensures that all connected devices on the enabled interfaces are
authenticated on the switch before they are allowed access. They need to use a pre-defined
password (either local or via RADIUS) that needs to match before the log-in is allowed on
the switch.
The port-security feature is the first feature that is configured. The initial task is using a specific
feature called auto-learning on port-security. This means that all devices that are connected
to the VSAN are automatically added to the port-security database and saved.
The port-security database can be distributed throughout the Fibre Channel fabric as well, so all autolearned entries are saved to all switches.
In the initial task you are not aware of the WWNs in the fabric so you will require using the autolearning feature.
MDS1
MDS1(config)# port-security enable
MDS1(config)# port-security auto-learn vsan 10
Error for VSAN 10: No Active database
MDS1(config)# port-security activate vsan 10
Now we enabled the port-security in VSAN 10 with auto-learning enabled. The next step is to
ensure that it is seen in the configuration of the device. This requires a command on the operational
prompt to copy the auto-learnt entries into the configuration.
MDS1
MDS1(config-if)# do sh port-security data v 10
------------------------------------------------------------------------------VSAN Logging-in Entity
Logging-in Point
(Interface)
------------------------------------------------------------------------------MDS1(config-if)# end
MDS1# port-security database copy vsan 10
MDS1# sh port-sec data
261

Logging-in Point
(Interface)
------------------------------------------------------------------------------10
c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10
c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10
10
c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10
20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)*
10
20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)*
[Total 6 entries]
MDS1#
In the configuration you will find all entries now.

MDS1# sh run vsan 10
version 3.3(1c)
vsan database
vsan 10
fcdomain fcid database
vsan 10 wwn 20:03:00:0d:ec:6f:4f:40 fcid 0x200000 area dynamic
vsan 10 wwn 20:04:00:0d:ec:6f:4f:40 fcid 0x200100 area dynamic
vsan database
port-security database vsan 10
pwwn c6:81:00:11:0d:17:fa:ce interface fc1/4
pwwn c6:81:00:11:0d:02:01:07 interface fc1/4
pwwn c6:80:00:11:0d:07:fa:ce interface fc1/3
pwwn c6:80:00:11:0d:01:01:07 interface fc1/3
swwn 20:01:54:7f:ee:05:08:79 interface fc1/5
swwn 20:01:54:7f:ee:05:08:79 interface fc1/6
port-security activate vsan 10
This copy action is a manual action. It makes it very easy for the initial roll-out of port-security to
have it enabled first and then disabled after the initial learning. Then you can add and remove things in
the database manually.
In this case there is no reason to disable the auto-learning so we leave it enabled.
Dont forget to perform this same action on MDS2, as there is no distribution enabled in this
question.
MDS2
MDS2(config)# port-security enable
MDS2(config)# end
MDS2# port-security database copy vsan 10
After its enabled we see the auto learned entries showing up in our database.
262
MDS2# sh port-sec data

Logging-in Point
(Interface)
------------------------------------------------------------------------------10
21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78(fc1/1)*
10
20:00:00:0d:ec:6f:4f:40(swwn) 20:05:54:7f:ee:05:08:78(fc1/5)*
10
20:00:00:0d:ec:6f:4f:40(swwn) 20:06:54:7f:ee:05:08:78(fc1/6)*
[Total 3 entries]
MDS2#
The next task will prepare you for new devices that will be connected into the fabric. The port-security
feature also knows the concept of wild-cards. You can enable certain characteristics of a host in the
database where you are unsure on which interface it will be connected. This means you can leave it
completely blank or you can add only a switch where it is added.
As we dont have any distribution of the database entries between the MDS switches, we do have an
overlap in this question as one of the PWWNs can be connected to either MDS switch.
MDS1
MDS1(config)# port-sec database vsan 10
MDS1(config-port-security)# ?
Port Security Lists:
any-wwn
Any Node/Port/Switch WWNs
device-alias N_Port Device_Alias
do
EXEC command
end
Exit from configure mode
exit
Exit from this submode
no
nwwn
N_Port Node-WWN
pwwn
N_Port Port-WWN
swwn
Switch-WWN
MDS1(config-port-security)# pwwn 20:00:00:a3:bf:33:11:33 ?
fwwn
Swith-Port WWN
interface Switch Interface name
swwn
Switch WWN
<cr>
Carriage return.
MDS1(config-port-security)# pwwn 20:00:00:a3:bf:33:11:33 interface fc1/11
MDS1(config-port-security)# pwwn 20:00:00:a3:fe:00:98:32
After the 2 entries have been added we see in the database that the entries area added and that for the
entry which has an interface configured the SWWN of MDS1 is added to it.
MDS1(config)# do sh port-sec database
Logging-in Point
(Interface)
------------------------------------------------------------------------------10
10
c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10
263
10
c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10
20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)*
10
20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)*
10
20:00:00:a3:bf:33:11:33(pwwn) 20:0b:00:0d:ec:6f:4f:40(fc1/11)*
10
20:00:00:a3:fe:00:98:32(pwwn)
Any
[Total 8 entries]
MDS1(config)# do sh wwn sw
Switch WWN is 20:00:00:0d:ec:6f:4f:40
MDS1(config)#
Now we need to configure MDS2 for the remaining PWWNs.

MDS2
MDS2(config)# port-security data vsan 10
MDS2(config-port-security)# pwwn 20:00:00:a3:fe:00:98:32
MDS2(config-port-security)# do sh wwn sw
Switch WWN is 20:00:54:7f:ee:05:08:78
MDS2(config-port-security)# pwwn 20:00:00:A3:DE:11:66:2B swwn
20:00:54:7f:ee:05:08:78
Verification about the just configured parameters. Its shown that one of the PWWNs can only be
connected to the SWWN of MDS2 and the other entry can be connected anywhere in the fabric.
MDS2(config-port-security)# do sh port-sec database
Logging-in Point
(Interface)
------------------------------------------------------------------------------10
21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78(fc1/1)*
10
20:00:00:0d:ec:6f:4f:40(swwn) 20:05:54:7f:ee:05:08:78(fc1/5)*
10
20:00:00:0d:ec:6f:4f:40(swwn) 20:06:54:7f:ee:05:08:78(fc1/6)*
10
20:00:00:a3:de:11:66:2b(pwwn) 20:00:54:7f:ee:05:08:78(swwn)
10
20:00:00:a3:fe:00:98:32(pwwn)
Any
[Total 5 entries]
MDS2(config-port-security)#
The next question in this task is to secure VSAN 20 and to manually configure all port-security
parameters. This time all parameters need to be distributed between the switches, because we can only
change MDS1 to finish this task. When we need to be as specific as possible, it means that we need to
connect PWWNs to their respective interfaces. This is the most specific we can get in a port-security
environment.
We first need the FLOGI information of both of the switches so we can create a database using this
information.
MDS1
MDS1(config-port-security)# do sh flogi database vsan 20
-------------------------------------------------------------------------INTERFACE VSAN
FCID
PORT NAME
NODE NAME
264
-------------------------------------------------------------------------fc1/3
20
0x0a00e8 c6:80:00:11:0d:07:fa:ce
c5:9a:00:06:2b:01:00:07
fc1/3
20
0x0a00ef c6:80:00:11:0d:01:01:07
c5:9a:00:06:2b:01:00:07
fc1/4
20
0x0a01e8 c6:81:00:11:0d:17:fa:ce
c5:9a:00:06:2b:02:00:07
fc1/4
20
0x0a01ef c6:81:00:11:0d:02:01:07
c5:9a:00:06:2b:02:00:07
MDS2
MDS2(config)# sh flogi data v 20
------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
------------------------------------------------------------------------------fc1/1
20
0x0c0000 21:01:00:e0:8b:3c:c0:b5
20:01:00:e0:8b:3c:c0:b5
Now onto the configuration of our database.

MDS1
MDS1(config-port-security)# port-security database vsan 20
MDS1(config-port-security)# pwwn c6:80:00:11:0d:07:fa:ce interface
MDS1(config-port-security)# pwwn c6:80:00:11:0d:01:01:07 interface
MDS1(config-port-security)# pwwn c6:81:00:11:0d:17:fa:ce interface
MDS1(config-port-security)# pwwn c6:81:00:11:0d:02:01:07 interface
MDS1(config-port-security)# pwwn 21:01:00:e0:8b:3c:c0:b5 ?
fwwn
Swith-Port WWN
interface Switch Interface name
swwn
Switch WWN
<cr>
Carriage return.
MDS1(config-port-security)# pwwn 21:01:00:e0:8b:3c:c0:b5 swwn
20:00:54:7f:ee:05:08:78 interface fc1/1
MDS1(config)# port-security commit v 20
fc1/3
fc1/3
fc1/4
fc1/4
After activating the database on the MDS1, we can see an activated database on both MDS switches
with all necessary information for activating the security in VSAN 20.
MDS1
MDS1(config)# do sh port-security data v 20
Logging-in Point
(Interface)
265
------------------------------------------------------------------------------20
20
c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
20
20
c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
20
21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78*
[Total 5 entries]
MDS1(config)#
MDS2
Logging-in Point
(Interface)
------------------------------------------------------------------------------20
c6:80:00:11:0d:07:fa:ce(pwwn) 20:03:00:0d:ec:6f:4f:40*
20
c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40*
20
c6:81:00:11:0d:17:fa:ce(pwwn) 20:04:00:0d:ec:6f:4f:40*
20
c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40*
20
21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78*(fc1/1)
[Total 5 entries]
MDS2(config)#
The last entry on MDS2 shows that it sees the interface that is configured on MDS1 back on MDS2. Our
port-security with distribution works!
Next up is Fabric Binding! This technology is used to protect Fibre Channel fabrics of rogue
switches to log-in to the fabric. Its also possible to create this kind of protection by using Port
Security. You can specify all of the SWWNs that are allowed access to the Fibre Channel fabric.
However the fabric-binding technology is more useful for this, because it can also use the Domain ID
for securing the fabric.
The initial task is to secure VSAN 30 so only MDS1 and MDS2 are allowed. Because we should also add
the Domain IDs in this task, we should make these static as a reboot might change the Domain ID
of a switch, which might cause the fabric to reject the switches and you might end up with loosing points
for this section.
The configuration must be done from both switches as fabric-binding cannot be distributed
between switches.
MDS1
MDS1(config)# sh wwn sw
Switch WWN is 20:00:00:05:9b:70:5a:00
MDS1(config)# fabric-binding data vsan 30
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 domain 1
MDS1(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 domain 2
MDS1(config-fabric-binding)# exit
MDS1(config)# fabric-binding activate v 30
266
MDS2
MDS2(config)# sh wwn sw
Switch WWN is 20:00:54:7f:ee:09:3e:c8
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00 domain 1
MDS2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8 domain 2
After activating the fabric-binding database we can verify the changes.

MDS1
MDS1(config)# sh fabric-binding data
-------------------------------------------------Vsan
Logging-in Switch WWN
Domain-id
-------------------------------------------------30
20:00:00:05:9b:70:5a:00
0x1(1) [Local]
30
20:00:54:7f:ee:09:3e:c8
0x2(2)
[Total 2 entries]
MDS2
MDS2(config)# sh fabric-binding data
-------------------------------------------------Vsan
Logging-in Switch WWN
Domain-id
-------------------------------------------------30
20:00:54:7f:ee:09:3e:c8
0x2(2) [Local]
30
20:00:00:05:9b:70:5a:00
0x1(1)
[Total 2 entries]
MDS2(config)# sh int trunk vsan
Vsan 1 is up (None)
Vsan 1 is up (None)
MDS2(config)#
When the VSAN is still up on the trunk interfaces we know that the fabric-binding is properly
configured!
The final topic in this security task is the FC-SP feature. This feature enables the authentication of hostswitch and switch-switch interfaces. It combines a CHAP authentication with a Diffie-Hellman
exchange. The steps for configuring FC-SP can become quite complex as a lot of different combinations
are possible.
267
An important topic is the combination of interface modes. You can configure an interface with different
modes as it might be required that you configure an interface with optional authentication. The table
below demonstrates the different modes and what will happen when these 2 interfaces are connected
to each other.
On
Auto-Active
Auto-Passive
Off
On
Yes
Yes
Yes
Link down
Auto-Active
Yes
Yes
Yes
No authentication
Auto-Passive Yes
Yes
No authentication No authentication
Off
No authentication
Link down
The On and Off modes are simple modes that do what they tell. They enforce authentication or reject
it. The auto modes are there for optional authentication. There should be at least 1 active link when
both sides are on auto as the passive link will wait for an authentication request before starting any
negotiation.
Behind the mode you have the option to configure the re-authentication timeout, so the authentication
is performed a couple of times and is not done only once in the lifetime of the interface.
We start by configuring the most powerful Diffie-Hellman group exchange.
MDS1
MDS1(config)# feature fcsp
MDS1(config)# fcsp dhchap dhgroup 4
MDS2
MDS2(config)# feature fcsp
MDS2(config)# fcsp dhchap dhgroup 4
The next step is to configure the password that you want to accept on the switch for authentication
requests. This can be configured as one key per peer device.
This means that when you have several devices connected to the switch you can configure 1 key per
connected device, by specifying the WWN of the switch.
In our example we need to be as specific as possible, although we only have 2 devices, we will still
configure that this key is only accepted for that particular MDS switch that is connected.
268
MDS1
MDS1(config)#
devicename
dhgroup
hash
preference)
password
fcsp dhchap ?
Configure password of the remote devices
Configure DHCHAP Diffie-Hellman Group List (In preference)
Configure DHCHAP Hash Algorithm List (In order of
Configure local DHCHAP password
MDS1(config)# fcsp dhchap password IPexpertMDS1 ?

<CR>
<hh:hh:hh:hh:hh:hh:hh:hh> WWN of the device with which to use this
local
password
MDS1(config)# fcsp dhchap password IPexpertMDS1 20:00:54:7f:ee:09:3e:c8
MDS2
MDS2(config)# fcsp dhchap password IPexpertMDS2 20:00:00:05:9b:70:5a:00
The password command is used for the accepted password on the local switch, whereas the
devicename command will configure the password that is used to authenticate the neighbor, so
the sending password.
Of course we need to configure this as well.
MDS1
MDS1(config)# fcsp dhchap device 20:00:54:7f:ee:09:3e:c8 password
IPexpertMDS2
MDS2
MDS2(config)# fcsp dhchap devicename 20:00:00:05:9b:70:5a:00 password
IPexpertMDS1
Now the passwords are configured and we can start the first authentication. MDS2 should not initiate
any request, so we will make this a passive link. MDS1 should set the link down when the switch
doesnt respond. As described in the table above, the only option that makes a link go down is the
on/off modes. So we are going to use this in this first task.
MDS1
MDS1(config)# interface fc1/1
MDS1(config-if)# fcsp on ?
<CR>
<0-100000> Provide reauthentication interval in minutes
MDS1(config-if)# fcsp on 15
MDS2
269
MDS2(config-if)# fcsp auto-passive

MDS2(config-if)#
When the FC-SP protocol is configured and the link is re-initialized, the authentication is working!
MDS1
MDS1(config-if)# sh fcsp interface fc1/1
fc1/1:
fcsp authentication mode:SEC_MODE_ON
reauthentication timeout (in minutes):15
Status:Successfully authenticated on Thu Oct 18 11:57:28 2012
Authenticated using local password database
MDS2
MDS2(config-if)# sh fcsp interface fc1/1
fc1/1:
fcsp authentication mode:SEC_MODE_AUTO_PASSIVE
Status:Successfully authenticated on Thu Oct 18 11:56:20 2012
Authenticated using local password database
The next link to authenticate needs to use SHA1 hashes with a fall-back to MD5. By default this is the
other way around. MD5 hashes are used by default with a fall-back to SHA1.
These are changed for the entire switch at the same time. We will configure both switches to respond
actively to authentication requests.
MDS1
MDS1(config-if)# fcsp dhchap hash sha1 md5
MDS1(config-if)# fcsp auto-active
MDS2
MDS2(config-if)# fcsp dhchap hash sha1 md5
MDS2(config-if)#
On the third link, which is the second member of port-channel 101 (fc1/3) we will completely
disable FC-SP, which is an easy thing to do. Keep in mind that ports have FC-SP auto-passive
enabled by default!
MDS1
MDS1(config-if)# fcsp off
MDS2
MDS2(config-if)# fcsp off
270
MDS2(config-if)#
And finally we configure authentication on the last link between the switches. One side can be
configured passive and one side as active. They will still come online when neither link authenticates,
but we do want authentication to happen. Therefore we need at least one end to be active.
MDS1
MDS2
MDS2(config-if)# fcsp auto-passive
MDS2(config-if)#
FC-SP is always configured under a port-channel member interface and never on the port-channel
itself!
Other places where FC-SP can be configured is on FCIP interfaces. FC-SP is never VSAN dependent.
Task 6: Advanced Features

The final task of this chapter is regarding some advanced Fibre Channel features. The CCIE Data Center
test is not made to test on a lot of advanced Fibre Channel MDS related features, but still you should be
aware with some of the core features.
The initial task is about configuring the distribution of features between switches. This distribution is
done using a Cisco proprietary protocol called Cisco Fabric Services (CFS), which has been
talked about earlier. Its possible with CFS to configure so-called regions. The regions that are
configured mean that you can create some MDS switches that have different configuration sets than
other MDS switches.
This is done by configuring the features in their own matching region number. By default every
feature resides in region 1. Up to 200 different regions can be configured.
In this case we want MDS2 to have its own call-home features. We can easily accomplish this by
configuring the call-home feature in a different region. When we then configure call-home (or
any other feature that uses CFS) on MDS1 and the changes are committed, they are not accepted on
MDS2 as a valid configuration and not inserted into the configuration.
271
The configuration for this feature is relatively easy, but it could introduce a lot of complexity in the MDS
configuration.
MDS2
MDS2(config)# cfs region 100
MDS2(config-cfs-region)# callhome
WARNING: If an Application is moved/assigned to a new region,
its scope is restricted to that region and it ignores all other regions
for distribution or merge.
Are you sure? (y/n) [n] y
The next feature that we need to configure are 2 things at the same time. The MDS switches can create
a list of SCSI targets that you can use in the entire FC fabric. This list can then be handed to the
Storage administrators to demonstrate that the FC network is working fine.
The next feature where we want to use this SCSI target list is a Job Scheduler. The job
scheduler is a unique feature of the NX-OS where you can schedule all CLI configurations to be done
either at a given date and time or by configuring it on a repetitive basis, like in our case where we need
to configure the scheduler to run every 24 hours.
The configuration needs to get used to, but there are no complex technologies involved in this case.
The initial result of the SCSI discovery is shown below.
MDS1# discover scsi-target local os all
discovery started
MDS1# show scsi-target devices
------------------------------------------------------------------------------VSAN
FCID
PWWN
VENDOR
MODEL
REV
------------------------------------------------------------------------------10
0xc500e8
c2:10:00:11:0d:0b:fa:ce
SANBlaze VLUN FC RAMDisk
4.0
10
0xc500ef
c2:10:00:11:0d:01:01:11
DGC
RAID 5
4.0
10
0xc501e8
c2:11:00:11:0d:1b:fa:ce
4.0
10
0xc501ef
c2:11:00:11:0d:02:01:11
DGC
RAID 5
4.0
MDS1#
Now we want to save this to the flash disk on the MDS switch. We can do this by using a Linux like
command.
MDS1# show scsi-target devices > VSAN10hosts.txt
MDS1# dir
557
Oct 18 12:22:05 2012 VSAN10hosts.txt
25
Jan 28 00:41:03 2012 cpu_logfile
272
20676608
49152
15668059
5596002
20576768
20676608
99152951
18954240
24628
101471413
Oct
Oct
Jan
Jan
Jan
Oct
Jan
Oct
Oct
Oct
02
02
28
28
28
02
28
02
18
02
19:06:38
19:51:56
00:43:53
00:48:17
00:45:33
17:07:43
00:48:01
17:11:33
11:06:14
17:13:42
2012
2012
2012
2012
2012
2012
2012
2012
2012
2012
kickstart-latest
lost+found/
m9000-ek9-ssi-mz.5.2.2a.bin
m9000-epld-5.2.1.img
m9200-s2ek9-kickstart-mz.5.2.2a.bin
m9200-s2ek9-kickstart-mz.5.2.6.bin
m9200-s2ek9-mz.5.2.2a.bin
m9200-s2ek9-mz.5.2.6.bin
mts.log
system-latest
-* Some filesystem space might be used for internal purposes. *

Usage for bootflash://
355192832 bytes used
337154048 bytes free
692346880 bytes total
MDS1# show file bootflash:VSAN10hosts.txt
------------------------------------------------------------------------------VSAN
FCID
PWWN
VENDOR
MODEL
REV
------------------------------------------------------------------------------10
0xc500e8
c2:10:00:11:0d:0b:fa:ce
4.0
10
0xc500ef
c2:10:00:11:0d:01:01:11
DGC
RAID 5
4.0
10
0xc501e8
c2:11:00:11:0d:1b:fa:ce
4.0
10
0xc501ef
c2:11:00:11:0d:02:01:11
DGC
RAID 5
4.0
MDS1#
Now we can schedule the task using our task scheduler.

MDS1(config)# feature scheduler
MDS1(config)# scheduler job name SCSI
MDS1(config-job)# discover scsi-target local os all
MDS1(config-job)# show scsi-target devices > VSAN10hosts.txt
MDS1(config-job)# exit
MDS1(config)# scheduler schedule name SCSI_SCHED
MDS1(config-schedule)# job name SCSI
MDS1(config-schedule)# time ?
daily
Specify a daily schedule
monthly Specify a monthly schedule
start
Specify a future time schedule
weekly
Specify a weekly schedule
MDS1(config-schedule)# time daily 00:00
MDS1(config-schedule)# exit
To verify we can check if the schedule is configured correctly and if the commands that we configured
under the job are the correct commands.
273
MDS1(config)# do sh scheduler job

Job Name: SCSI
-------------discover scsi-target local os all
show scsi-target devices > VSAN10hosts.txt
==========================================================================
====
MDS1(config)# do sh scheduler schedule
Schedule Name
: SCSI_SCHED
-------------------------------User Name
: admin
Schedule Type
: Run every day at 0 Hrs 0 Mins
Last Execution Time : Yet to be executed
----------------------------------------------Job Name
Last Execution Status
----------------------------------------------SCSI
-NA==========================================================================
====
MDS1(config)#
The final 2 tasks of this chapter are introducing another unique Cisco MDS feature.
We are asked to create failovers between devices where the end-user can never notice any changes in
the Fibre Channel network.
The feature that supports this is SAN Device Virtualization (SDV). This feature is a hidden
command to enable since NX-OS 5.x, but it could very well still be an option to configure on the CCIE
Data Center test.
Once the feature is enabled, all commands are still available like they used to be.
We can create a virtual Fibre Channel device on our network that in the background has 2 devices
connected to it for virtualization purposes. We can configure automatic failover and fallback for this
physical device.
The virtual device is taken into the zoning. So we should create a zone for this use so the users in VSAN
10 can communicate to it.
We configure SDV with the following configuration.
MDS1
MDS1(config)# sdv virtual-device name JBOD1 vsan 10
MDS1(config-sdv-virt-dev)# device-alias J1D1 primary
MDS1(config-sdv-virt-dev)# device-alias J2D1
MDS1(config-sdv-virt-dev)# attribute ?
failover Configure the failover attributes
MDS1(config-sdv-virt-dev)# attribute failover ?
auto Enables Auto-failover for a virtual-device
274
MDS1(config-sdv-virt-dev)# attribute failover auto ?

<CR>
fallback Enables Switchback with Auto-failover
MDS1(config-sdv-virt-dev)# attribute failover auto fallback
One important note is the attribute command where we configure the actual automatic failover and
fallback. By default the SDV device will keep pointing to the PWWN of the primary device, unless the link
command is used to change this. We can make this process automatic, by configuring the automatic
failover and fallback parameters.
When the SDV configuration is committed, a device-alias is automatically created for use within the
zoning.
MDS1(config)# sdv commit vsan 10
MDS1(config)# sh device-alias data
device-alias name JBOD1 pwwn 50:00:53:00:01:03:d0:0a
MDS1(config)#
You successfully completed chapter 5 regarding Cisco MDS Fibre Channel features. The next chapter will
focus on Fibre Channel extension over IP connections!
Do NOT erase configurations on the MDS switches before moving on to the next Chapter!
275
Chapter 6: Data
Center Storage
Networking
Extension
Chapter 6: Data Center Storage networking Extension is intended to let you be familiar with the
Storage Networking features on the Cisco MDS switches. This chapter will be about configuring IP
features like iSCSI, iSLB and FCIP including the relevant Security features for Fibre Channel extension
across IP connections. We highly recommend to create your own diagram at the beginning of each lab
so you are able to draw on your own diagram, making it much easier when you step into the real lab.
276
General Rules
particular chapter
5 hours
277
Solutions
Task 1: Initial set-up
In this chapter we will be configuring several IP features related to the Cisco MDS platform. The
MDS platform has some unique IP features that no other Fibre Channel switch vendor has.
Initially Cisco offered several line-cards in the chassis based MDS switches, which supported
Ethernet (1Gbps) interfaces which have these features. In the current generation platforms
(MDS9222i), these interfaces are integrated in the switch.
On the IP features the MDS platform has 2 strong and prominent features, which are tested, in
the CCIE Data Center:
Fibre Channel over IP (FCIP) tunnels
With FCIP tunnels you are able to extend the Fibre Channel network across a normal
Layer 3 IP network. This overcomes the distance limitations that you have when
expanding Fibre Channel between data centers, where you could easily run into issues
with B2B credits. Besides that you need a Dark Fiber or DWDM transparent system to
transport Fibre Channel. FCIP overcomes these limitations and narrows down your
requirements for transporting FC between Data Centers to just an IP connection
iSCSI
The iSCSI feature on the Cisco MDS platform is not widely used, but still a great
feature and most important of all tested on the CCIE Data Center lab exam. This feature
makes it able for you to present a Fibre Channel disk to an non-FC host. Meaning that
the Server can talk over its IP connection with the iSCSI protocol to the MDS switch
which will present itself as an iSCSI LUN, while in the background you are actually
talking to a Fibre Channel disk.
The iSLB feature is a extension of the iSCSI feature where redundancy, distribution
and load balancing are integrated. You could say iSLB is actually iSCSI 2.0 on the
MDS platform
Be aware that you need the SAN Extension over IP license on the MDS switches to enable
FCIP. For iSCSI you do not require a license. On the MDS 9222i the SAN Extension
license is included!
For this task we will be required to use a Ethernet switch to transport our traffic between our
MDS switches. Our physical topology shows that we have our MDS IP ports connected to the
Nexus 5500 switches. This means that we first need to ensure that we transport several VLANs
between the switches.
278
We will be using several VLANs during this chapter, which is not strictly necessary, but do let
you practice with the 802.1Q features of the IP interfaces on the line cards. The IP interfaces of
the MDS switch are different than a standard router or switch interface. Keep in mind that
these interfaces are actually just host interfaces that let you connect a host device to it. This
means that you can just configure an IP address in the same subnet on multiple interfaces. As
they are just hosts on the network and will never start routing between these ports.
The MDS also support port-channeling of these interfaces in standard Ethernet portchannels. In this chapter we will not be configuring Ethernet Port-channels on the MDS switch,
but configuration is equal to how that is done on the Nexus switches.
Drawing 1 shows you the physical topology we are using in this chapter.
In the first task we are required to ensure that the MDS switches can communicate over a
number of VLANs. You might still have configuration from a previous task on the Nexus
switches, but this is not conflicting with this lab.
The first task is to configure the VLANs and let the MDS switches have access to it. This means
we need to make sure that both GigabitEthernet interfaces on the MDS switches are configured
as Trunk interfaces on the Nexus 5500s.
Configuration is a piece of cake, as we have already finished extensive chapters regarding the
configuration of NX-OS with Ethernet and IP features.
279
Configuration in this case will only comprise of the VLANs and Trunk configurations. We do not
need to configure IP routing, where this would be an option for FCIP and iSCSI as well. In this
case it would not add a lot to our complexity and we just want to separate traffic across
different VLANs.
We use Drawing 2 to determine the VLANs that we need.
In this drawing we can see that we should be using VLAN 69 through 72 for the IP features of
the MDS switches. In this chapter we will not be using any ISL links between the MDS switches.
We have pre-configuration from the previous chapter on there, but this is not conflicting with
the tasks that we are about to configure.
We re-configure the interfaces between the MDS switches as we require some other traffic
than FabricPath from the last Chapter where we configured the Nexus switches.
SW2
SW2(config)# vlan 69,70,71,72
SW2(config-if)# switchport mode trunk
SW2(config-if)# switchport trunk allowed vlan 69-72
280

SW3
SW3(config)# vlan 69,70,71,72
MDS1
MDS1(config)# interface gig1/1
MDS1(config-if)# exit
MDS2
We do not need to configure any IP addressing right now. The final statement in this task is
regarding the future configurations of IP addressing, where we need to pay attention to the
format that we are going to use. Drawing 2 will illustrate which host IP addresses need to be
used and which VLANs belong to which feature.
So we finished the first task!
281
Task 2: FCIP
In the second task we will start configuring our FCIP tunnels between the switches. In this
chapter you will be configuring both the FCIP tunnels and their High Availability
features that are available on the Cisco MDS platform.
During your CCIE Data Center lab exam you might see questions about the expansion of your
Fibre Channel network to the other Data Center in the lab topology.
Keep in mind that its not possible to transport FCoE across the data centers when you are
using FabricPath and/or OTV. Therefore you will see MDS switches with FCIP features being
used in Cisco best practice designs regarding multiple Data Centers.
The FCIP technology is a standard based solution (FC-BB-2) and compatible with other
vendors FC switches.
The goal of the technology is to extend the Fibre Channel fabric across geographically dispersed
locations.
The only traffic that you are able to transport is E-port type traffic, meaning you can only
transport ISL traffic. The Virtual E-port (VE) that is created has the same features as a
normal E-port. Meaning that you can also enable the Trunking protocol to transport multiple
VSANs.
The FCIP interface is the same interface as a normal FC interface on the MDS switch, meaning
that you have an equal amount of configuration commands available.
The drawing below illustrates the positioning of the FCIP tunnel in the Fibre Channel fabric.
282
The Host and the JBOD in this drawing do not know that they are interconnected through a
Layer 3 IP network as they just talk plain Fibre Channel throughout the FC fabric. All 4 FC
switches are participating in the same FC Domain and FSPF network. Traffic is routed in the
same way, except that the traffic gets an IP encapsulation instead of an FC encapsulation
between the data centers.
The FCIP tunnel consists of 2 TCP sessions:
Control
The control session is used for all Fibre Channel control packets. This means that all
Class F fibre channel frames are transmitted across this TCP session. This ensures that
control traffic can receive a different QoS treatment throughout the IP network to
ensure a low latency transmission of all control frames.
Data
The second TCP session is used for all other data (Class 1, 2, 3) frames that are
sent across the FC fabric. These data packets can get another QoS marking so they fall in
different classes when required.
You can choose to have only 1 TCP session for the FCIP connection. By default there will be 2
sessions.
Configuration of the FCIP connection consists of 2 sections:
FCIP Profile
The FCIP profile configured the TCP session parameters and the local interface
parameters. This configures the source interface of the FCIP tunnel. A single profile
can be used by multiple tunnels.
FCIP Interface
The FCIP interface configurations is the actual tunnel interface. On this interface all
Fibre Channel related features are configured, a profile is attached, the peer IP
address is configured and the number of TCP sessions used for this tunnel.
To create a FCIP interface you need to have at least 1 differentiation out of these 4
parameters:
Local IP address
Local Listener Port
Peer IP address
Peer port
283
You can not create multiple interfaces where all these parameters are equal, as that would
result in the same interface. When one of these 4 is different, the interface can come up, but it
might require some engineering regarding the initiation of the interface.
Our first question of the task is to configure a FCIP connection between the 2 MDS switches.
We will transport only 2 already active VSANs across this interface. The FCIP tunnel is using
only a single interface and using only 1 TCP session between the switches.
We can not use the default TCP port, which is TCP 3225, so we need to change this as well. As
this is a TCP session parameter we need to configure this under the profile configuration.
MDS1
MDS1(config-if)# interface gig1/1.69
MDS1(config-if)# ip address 198.18.69.9 255.255.255.0
MDS1(config-if)#
MDS1(config-if)# feature fcip
MDS1(config)# fcip profile 1
MDS1(config-profile)# ip address 198.18.69.9
MDS1(config-profile)# port ?
<1-65535> Profile TCP Port
MDS1(config-profile)# port 3226
MDS1(config-profile)# tcp ?
cwm
Enable congestion window monitoring
keepalive-timeout
Set keep alive timeout in sec
max-bandwidth-kbps
Configure maximum available path bandwidth in Kbps
max-bandwidth-mbps
Configure maximum available path bandwidth in Mbps
max-retransmissions Maximum number of retransmissions
min-retransmit-time Set minimum retransmit time in millisec
pmtu-enable
Enable PMTU Discovery
sack-enable
Enable SACK option for TCP
send-buffer-size
Send buffer size in KBytes
MDS1(config-profile)#
The profile configuration now has the source IP address configured and the non-default port
that we are required to use. Other TCP parameters that can be set are traffic-shaping
parameters and some TCP features. All the advanced TCP features like Path MTU Discovery
(PMTU) and Selective Acknowledgements (SACK) are by default enabled.
284
The interface that we configure has a sub-interface number of 69. This number is equal to
the VLAN number, because this is immediately the VLAN configuration. There is no
encapsulation command like on normal router/switch interfaces. So keep in mind which
sub-interface number you use for this case as its crucial for the VLAN numbering.
The amount of TCP connections is still supported to be configured, except that its not available
through the question mark anymore. Therefore you should know that this command is
configured under the FCIP interface with the tcp-connections command. You may
specify a value of 1 or 2 behind it, where 2 is the default.
Now the FCIP interface should be configured.
MDS1
MDS1(config-profile)# interface fcip1
MDS1(config-if)# use-profile 1
MDS1(config-if)# peer-info ipaddr 198.18.69.11 port 3226
MDS1(config-if)# tcp-connections 1
MDS1(config-if)# switchport ?
description Enter description of maximum 80 characters
fcrxbufsize Configure receive data field size for the port
mode
Enter the port mode
trunk
Configure trunking parameters on an interface
MDS1(config-if)# switchport mode ?
E
E port mode
auto Autosense mode
MDS1(config-if)# sw trunk mode on
MDS1(config-if)# sw trunk allowed vsan 10
MDS1(config-if)# ?
channel-group
Add to/remove from a port-channel
fcdomain
Configure fcdomain parameters
ficon-tape-accelerator
Configure the FCIP FICON tape accelerator
ficon-tape-read-accelerator Configure FCIP FICON Read accelerator
ficon-xrc-accelerator
Configure FCIP FICON XRC accelerator
ficon-xrc-logging
Configure Ficon XRC Logging
fspf
Configure FSPF parameters
ip-compression
Configure the IP compression for the FCIP tunnel
link-state-trap
Enable/disable link state change traps
no
passive-mode
Fcip set passive connect mode
peer-info
Configure fcip peer information
qos
Configure qos for interface
shutdown
Enable/disable an interface
switchport
Configure switchport parameters
use-profile
Fcip use profile for tunnel
write-accelerator
Configure the FCIP write accelerator
end
Go to exec mode
exit
pop
push
where
285
MDS1(config-if)# vsan database
MDS1(config-vsan-db)# vsan 10 interface fcip1
The final configuration that is done in this case, adding the FCIP interface to a VSAN in the
VSAN database, has to do with the fact that one of the tasks in the previous chapter was to
deactivate the Default VSAN (VSAN 1). By making it suspended, every interface that has its
registration in VSAN 1 will stay down unless something else is configured. In the case of Trunk
interfaces, the port first needs to come online before the trunking is negotiated with the other
side. When the trunk port is configured in the Default VSAN it will first come online in this
VSAN before it will start trunking. Therefore we need to configure the FCIP interface, as its
treated as just another FC interface, inside the VSAN database into another VSAN. The VSAN
that you configure the tunnel in, is not that significant, except that it should be anything else
than VSAN 1.
The FCIP connection is now configured on one side. There is one additional requirement for
this initial few questions of this task, which is that MDS1 should always initiate the connection.
This is achieved using the passive-mode command, where the switch where its configured
will never initiate the connection and will always wait on the other side for setting up the
tunnel.
This passive-mode is required when you are configuring multiple FCIP tunnels that each
should terminate on the same FCIP profile, meaning you have only 1 destination IP/port for
the remote side. When this switch with only 1 listening port, would initiate the connections
there is no problem as the switch will initiate the connection correctly to the right switch. When
it receives traffic on the FCIP port, it will not know to which FCIP tunnel the traffic actually
belongs.
In this case it has no special use, except that you know how to configure this.
What remains is to configure the other side.
MDS2
MDS2(config-if)#
MDS2(config-if)# feature fcip
MDS2(config)# fcip profile 1
MDS2(config-profile)# ip address 198.18.69.11
MDS2(config-profile)# port 3226
MDS2(config-profile)# interface fcip1
MDS2(config-if)# peer-info ipaddr 198.18.69.9 port 3226
MDS2(config-if)# tcp-connections 1
MDS2(config-if)# sw trunk mode on
MDS2(config-if)# passive-mode
286

Now we need to verify our configuration.

First we verify our FCIP profile configuration.
MDS1(config-if)# sh fcip profile all
FCIP Profile 1
Internet Address is 198.18.69.6 (interface GigabitEthernet1/1.69)
Tunnels Using this Profile: fcip1
Listen Port is 3226
TCP parameters
SACK is enabled
PMTU discovery is enabled, reset timeout is 3600 sec
Keep alive is 60 sec
Minimum retransmission timeout is 200 ms
Maximum number of re-transmissions is 4
Send buffer size is 0 KB
Maximum allowed bandwidth is 1000000 kbps
Minimum available bandwidth is 500000 kbps
Configured round trip time is 1000 usec
Congestion window monitoring is enabled, burst size is 50 KB
Auto jitter detection is enabled
MDS1(config-if)#
As mentioned the advanced features like SACK and PMTU are by default enabled. The best
command to verify the working of all your FCIP interfaces is the following command. It
shows you a summary of all important parameters of the interfaces.
MDS1(config-if)# show fcip summary
------------------------------------------------------------------------------Tun prof
Eth-if
peer-ip
Status T W T Enc Comp Bandwidth
rtt
E A A
max/min
(us)
------------------------------------------------------------------------------1
1
GE1/1.69
198.18.69.11
UP
Y N N N
N
1000M/500M 1000
MDS1(config-if)#
The detailed information for each FCIP tunnel is found in a normal show interface.
MDS1(config-if)# show interface fcip1
fcip1 is up (Trunking)
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:54:a9:40
Port vsan is 1
Using Profile id 1 (interface GigabitEthernet1/1.69)
Peer Information
Peer Internet address is 198.18.69.11 and port is 3226
Write acceleration mode is configured off
Tape acceleration mode is configured off
Tape Accelerator flow control buffer size is automatic
FICON XRC Accelerator is configured off
Ficon Tape acceleration configured off for all vsans
IP Compression is disabled
Maximum number of TCP connections is 1
287

QOS
QOS
TCP
1
control code point is 0

data code point is 0
Connection Information
Active TCP connections
Local 198.18.69.9:65523, Remote 198.18.69.11:3226
0 host table full 0 target entries in use
6 Attempts for active connections, 3 close of connections
TCP Parameters
Path MTU 1500 bytes
Current retransmission timeout is 200 ms
Round trip time: Smoothed 8 ms, Variance: 4 Jitter: 157 us
Advertized window: Current: 34 KB, Maximum: 24580 KB, Scale: 5
Peer receive window: Current: 2047 KB, Maximum: 2047 KB, Scale: 5
Congestion window: Current: 14 KB, Slow start threshold: 112 KB
Current Send Buffer Size: 34 KB, Requested Send Buffer Size: 0 KB
CWM Burst Size: 50 KB
Measured RTT : 115 us Min RTT: 500000 us Max RTT: 143 us
18 Class F frames input, 2736 bytes
0 Class 2/3 frames input, 0 bytes
0 Reass frames
0 Error frames timestamp error 0
23 Class F frames output, 2916 bytes
0 Class 2/3 frames output, 0 bytes
0 Error frames
MDS1(config-if)#
Now multiple things are shown in this output. You can see the specifics of this particular FCIP
interface. This is separated in the information related to the TCP parameters of the session
and the Fibre Channel parameters of this interface. The FCIP interface is just a normal FC
interface when looked at from a Fabric perspective.
We can see in this output that we are indeed using 1 TCP connection and a non-default port,
which is initiated by MDS1, because it uses a destination port of 3226. When the switch
wouldve received the connection the source port would be 3226.
Now we have our first FCIP connection successfully up and running!
288
The next question in this task asks you to configure high availability for this FCIP
tunnel. We can solve this using 4 solutions.
Redundant FCIP tunnel
We could create an additional FCIP tunnel where the FCIP 1 tunnel can failover to
the other using a FSPF re-calculation. However we can not change the FCIP
configuration for this task.
Fibre Channel port-channel
As the FCIP tunnel is treated as a regular Fibre Channel interface, its also possible to
bundle the FCIP interfaces together in a FC port-channel which would be a
seamless switch over. However we can not change the FCIP configuration for this task
and we can not use port-channels.
Ethernet port-channels
We have the option to configure Gig1/1 and Gig1/2 in an Ethernet port-channel
which would solve our question with a seamless failover for the single FCIP 1
connection. However we can not use port-channels for this task.
VRRP
The last and only correct option is the use of the Layer 3 redundancy protocol VRRP.
This protocol is the only FHRP protocol available on the MDS switches. With the use of
this protocol we can still have a single IP address for both the source interface and the
peer IP address. When something happens when the first GigabitEthernet interface
the VRRP configuration will issue a failover to the other. As the GigabitEthernet
interfaces on the MDS switches are just treated as Host interfaces, its perfectly
possible to run a local VRRP on the same MDS switch.
We will perform the VRRP configuration in a way that we do not need to change the FCIP
configuration as we are not allowed to do this according to the question. This means that we
need to re-use our interface IP addressing in the VRRP group configuration.
When we need to create additional IP addresses in the subnet this would be perfectly allowed
according to the current question requirements!
Lets start by configuring the VRRP group using the correct IP addressing so we do not need to
change the FCIP interfaces for this change.
MDS1
MDS1(config-if)# vrrp 69
MDS1(config-if-vrrp)# address 198.18.69.9
MDS1(config-if-vrrp)# priority 120
MDS1(config-if-vrrp)# no shutdown
MDS1(config-if-vrrp)# exit
289

MDS2
MDS2(config-if-vrrp)# priority 120
Now with these VRRP groups configured we still have an active FCIP tunnel between the
originally configured endpoints as our endpoints didnt change. We did have to configure
additional IP addresses, but there is no statement in the question that prohibits you from using
additional IP addresses in the subnet to accomplish the question.
MDS1
MDS1(config-if-vrrp)# sh vrrp
Time Pre State
VR IP addr
--------------------------------------------------------------------------GigE1/1.69 69
IPv4
120
1 s
master 198.18.69.9
GigE1/2.69 69
IPv4
100
1 s
backup 198.18.69.9
MDS1(config-if-vrrp)#
MDS2
MDS2(config-if-vrrp)# sh vrrp
Time Pre State
VR IP addr
--------------------------------------------------------------------------GigE1/1.69 96
IPv4
120
1 s
master 198.18.69.11
GigE1/2.69 96
IPv4
100
1 s
backup 198.18.69.11
290
Keep the VRRP group numbers different on the 2 MDS switches. You are configuring 2 VRRP
groups in the same VLAN. Remember that the group number is used in the virtual MAC
address of this virtual default gateway. When you configure equal group numbers
in the same VLAN, this will cause serious issues and an unworkable situation. The VRRP
configuration on both DMS switches should work fully independent of each other. Therefore
you will need to use different group numbers on both MDS switches.
We see that the local MDS switch is backup for itself. Therefore when something happens with
GigabitEthernet1/1 the second interface will become the master and the FCIP
connection
remains
up!
The next task will create the second FCIP connection between the MDS switches by using
another interface. Again we need to take care of a few specifics in this FCIP connection. When
you read ahead on the next number of questions you will see some specifics that need to be
applied to this connection.
For example, we need to ensure that this FCIP 2 connection receives a higher QoS priority.
Well we just saw in the verification of FCIP 1 that the default QoS marking is 0. So anything
higher than this is good for a better treatment in the IP network, when properly configured.
The second point is that the tunnel must be brought down when there are no packets received
for 90 seconds. This feature is the tcp keepalive-timeout configured in the FCIP
profile, as its a TCP parameter.
MDS1
MDS1(config-if)#
MDS1(config-if)# fcip profile 2
MDS1(config-profile)# ip add 198.18.70.9
MDS1(config-profile)# tcp ?
cwm
keepalive-timeout
max-bandwidth-kbps
max-bandwidth-mbps
Configure maximum available path bandwidth in Mbps
pmtu-enable
sack-enable
send-buffer-size
MDS1(config-profile)# tcp keepalive-timeout ?
<1-7200> TCP Keep Alive timeout value
MDS1(config-profile)# tcp keepalive-timeout 90
MDS1(config-profile)# ?
ip
Config ip to profile
no
port
Config local port to profile
tcp
Config TCP Parameters for the Profile
end
Go to exec mode
exit
pop
291

push
where

MDS1(config-profile)# exit
MDS1(config)# interface fcip 2
MDS1(config-if)# peer ip 198.18.70.11
MDS1(config-if)# qos ?
control Configure qos for control and data packets
MDS1(config-if)# qos control ?
<0-63> QOS control code point value
MDS1(config-if)# qos control 15 ?
data Configure qos for control and data packets
MDS1(config-if)# qos control 15 data 10
MDS1(config-if)# sw mod e
We first create a new sub-interface on Gig1/2 with the new VLAN. We could have configured
this FCIP connection in the same VLAN as we are using different port numbers in the first
VLAN. So using the default port numbers would generate enough separation in the profile, as
discussed at the start of this task, to create an additional FCIP connection in the same VLAN.
In this case we create a new VLAN with different IP addressing. In the FCIP profile we
configure the keepalive-timeout to be 90 seconds, so the tunnel is brought down after
this time out expires.
Then under the FCIP interface we configure the QoS values for this interface. The requirement
is that it should be higher than the FCIP 1 interface. In this example you see that its possible
to create a different QoS mapping for both the Control session and the Data session.
Requirement for this to work is that you always use the 2 TCP connection configuration, which
is the default.
Finally we need to add this second FCIP connection toe the VSAN database as our Default
VSAN 1 is suspended, which means the interface will not come online when it is configured in
that VSAN.
Now we need to bring the connection up on the second MDS.
MDS2
MDS2(config-if)#
292

When both FCIP 1 and FCIP 2 are up and running we can configure the load balancing
parameters that we need to comply with.
We need to ensure that VSAN 10 traffic uses one link and VSAN 20 the other link. From the
standpoint of MDS2 this should be vice versa. This means that on MDS2 the traffic for VSAN 10
should use the FCIP 2 connection and VSAN 20 traffic should be using the FCIP 1
connection.
This means that we will be using the FSPF costs to alter the behavior of where traffic uses
which link. The FCIP interfaces are by default seen as 1Gbps links. We need to use these links
as the primary links for VSAN 10 and 20, as this is stated by the questions. Keep in mind to
lower the costs to ensure that this happens. As said the FCIP links are seen as 1Gbps links,
meaning they have a default FSPF cost of 1000.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
interface
fspf cost
fspf cost
interface
fspf cost
fspf cost
fcip1
10 vsan
20 vsan
fcip2
20 vsan
10 vsan
interface
fspf cost
fspf cost
interface
fspf cost
fspf cost
fcip1
20 vsan
10 vsan
fcip2
10 vsan
20 vsan
10
20
10
20
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
10
20
10
20
With these changes the traffic will now flow across the correct FCIP links.
The next question in this task is regarding compression. FCIP connections can compress their
data to ensure a more optimal way of sending traffic across the links.
293
In the past there where 3 modes of compression. This has been changed to only 2 modes:
Auto mode
This is enabled by default and will compress files when necessary and when the other
side of the FCIP connection negotiates compression as well.
Mode 1
This mode is used for higher bandwidth links, so less compression. Dont let yourself go
blind on the description you get from the CLI. As that will tell you that mode 1 is High
compression, but they actually mean its High Speed compression compared to Mode 2.
Mode 2
This mode enables a higher compression rate, but introduces latency due to the
compression algorithm that takes time to run.
As we want the highest compression as possible in this question we need to configure Mode 2
on both sides. Dont let yourself be confused with the Mode 1 CLI comment!
MDS1
MDS1(config-if)# interface fcip2
MDS1(config-if)# ip-compression ?
<CR>
auto
Auto compression setting
mode1 High compression
mode2 Moderate compression for medium bandwidth links
MDS1(config-if)# ip-compression mode2
MDS1(config-if)#
MDS2
MDS2(config-if)#
The compression modes need to match on both sides otherwise you could run into issues with
the FCIP tunnels. Auto mode will solve all these issues.
The next question in the task wants that FCIP 1 sends R_RDY messages locally. Which means
that the messages are sent by the local MDS where they should actually be sent by the remote
host. When a host receives a confirmation a packet is arrived it will close the write action on its
disk and assumes everything is fine. Now this responsibility comes in the hands of the MDS
switch.
294
If the MDS switches will send confirmations for write actions this means that the disks will
respond faster and will finish their actions faster.
This technology is called write-accelerator. It can be configured for both normals disks
and tapes. For tape traffic a little different style of acknowledgements is sent and received, so
this is adapted for by the MDS switch.
Its easy to enable by using a single command. Ensure you have enabled this on both sides of
the connection, otherwise traffic could behave strangely.
MDS1
MDS1(config-if)# write-accelerator
MDS2
MDS2(config-if)# write-accelerator
The final question of this task asks to configure high availability for connection FCIP 2. We
should do this using a third FCIP connection and we can not use Ethernet port-channels for the
high availability.
This leaves only one option which is configuring a Fibre Channel based port-channel
configuration with FCIP connections.
We need to ensure that the tasks that we configured before this will be kept in the portchannel configuration so we dont loose any points there.
The task mentions that we should keep high availability in mind when we are configuring this
question. To ensure we have the most resiliency on this third FCIP connection we should use a
different Ethernet interface as well, which we can easily do.
We can use the GigabitEthernet1/1 connection in VLAN 70, as we havent configured that
interface yet. We will need to use different IP addressing in this case as we share the same
VLAN. The question does not prohibit the creation of additional IP addresses in the subnet, so
we can do this.
Because we are creating a Fibre Channel port-channel the FSPF protocol does not notice the
change when one interface is lost in the port-channel FSPF costs will need to remain the same
for the configured VSANs in the previous question.
MDS1
MDS1(config-if)#
295

MDS1(config-if)#
MDS2
MDS2(config-if)#
MDS2(config-if)#
Now we configured the third FCIP connection across the other GigabitEthernet interface
so we have high availability for those 2 FCIP connections.
To finish we need to force the tunnels to form a port-channel. We need to ensure we keep
the configuration of FCIP 2 leading for the port-channel.
MDS1
MDS1(config-if)#
MDS1(config-if)# fspf cost 20 vsan 10
296
MDS2
MDS2(config-if)#
Now the port-channel has formed and we have 2 members configured in this port-channel. It
transports all the VSANs that FCIP 2 does and it has the same interface properties like FSPF
costs for the VSANs and we have configured IP-Compression like we did on FCIP 2.
Finally dont forget to add this connection to the VSAN database as well, otherwise it will
remain down.
We now have a high available FC port-channel connection for FCIP!
Task 3: FCIP Security
The third task is regarding the security of our FCIP connections. We will be configuring 2
different security mechanisms in this task. The second mechanism is used widely in the world
for connections between different networks. Therefore its not strange that this technology is
also found in the MDS switch to protect the IP traffic that it needs to send to different data
centers in the world. This technology is IPsec together with IKE.
Not all MDS switches support this technology, but the ones used in the lab do. Security features
are tested so be prepared for this configuration.
The first task asks us to secure the failover mechanism of FCIP 1. This means we have to
protect our VRRP with a MD5 hash. We already did this once in the High Availability
chapter so this is not a complex configuration.
The solution is just a few commands and we have a secure protocol between our local
interfaces. Dont forget to configure both MDS switches!
MDS1
297

MDS1(config-if-vrrp)# authentication ?
md5
Set the MD5 authentication key (16 char max)
text Set the authentication password (8 char max)
MDS1(config-if-vrrp)# authentication md5 SecureIPexpert
^
% Incomplete command at '^' marker.
MDS1(config-if-vrrp)# authentication md5 SecureIPexpert ?
spi Security parameter index
MDS1(config-if-vrrp)# authentication md5 SecureIPexpert spi ?
<0x0-0xffffffff>
MDS1(config-if-vrrp)# authentication md5
The vr configuration can only be updated
the operational state is initialize
MDS1(config-if-vrrp)# shut
MDS1(config-if-vrrp)# no shut
SecureIPexpert spi 0x0

when the administrative state is down and
One thing different about configuration of authentication in VRRP on the MDS switches is that
you need to configure a Security Parameter Index (SPI). This is a unique key for the
authentication key that is communicated to the neighbor. This SPI should be unique for each
VSAN!
MDS2
MDS2(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x1
MDS2(config-if-vrrp)# authentication md5 SecureIPexpert spi 0x1
298
After this configuration the communication of the VRRP is now authenticated between the
neighbors.
The next 2 tasks are regarding the security of the FCIP 1 connection. This configuration is
quite complex on the MDS switch and you should pay close attention to what you configure
here.
The IPsec feature is not available on all platforms, but it is available on the platform in the CCIE
Data Center lab topology, MDS 9222i. Its not available on the IPS-4 and IPS-8 line cards.
You also require a ENTERPRISE license for this feature.
Configuration is similar to the IOS configuration for IPsec VPNs, but you will find a little
different commands are used here and there.
The question states that we should secure the FCIP 1 connection. Keep in mind that you use
the VRRP configuration as the peer address configurations, but you should apply the IPsec
tunnels to both interfaces where the tunnel could terminate. When this isnt done, you will
loose points as the FCIP 1 connection is high available and your security not!
Lets start the configuration on MDS1.
MDS1
MDS1(config)# feature crypto ike
MDS1(config)# feature crypto ipsec
MDS1(config)#
MDS1(config)# crypto ike domain ipsec
MDS1(config-ike-ipsec)# identity address
MDS1(config-ike-ipsec)# key IPexpertEncrypt address 198.18.69.11
MDS1(config-ike-ipsec)# policy 1
MDS1(config-ike-ipsec-policy)# encr aes
MDS1(config-ike-ipsec-policy)# hash md5
MDS1(config-ike-ipsec-policy)# authentication pre-share
MDS1(config-ike-ipsec-policy)# group 2
MDS1(config-ike-ipsec-policy)# exit
MDS1(config-ike-ipsec)#
MDS1(config-ike-ipsec)# ip access-list FCIP1IPSEC permit ip 198.18.69.9 0.0.0.0
198.18.69.11 0.0.0.0
MDS1(config)# crypto transform-set domain ipsec FCIP1 esp-aes ?
128 ESP transform using AES 128 bit cipher
256 ESP transform using AES 256 bit cipher
MDS1(config)# crypto transform-set domain ipsec FCIP1 esp-aes 128 esp-md5-hmac
MDS1(config)# crypto map domain ipsec FCIP 1
MDS1(config-crypto-map-ip)# match address FCIP1IPSEC
MDS1(config-crypto-map-ip)# set peer 198.18.69.11
MDS1(config-crypto-map-ip)# set transform-set FCIP1
MDS1(config-crypto-map-ip)# interface gig1/1.69
MDS1(config-if)# crypto map domain ipsec FCIP
299
We configured the IPsec tunnel on both interfaces for only the traffic that is using the VRRP
group addresses. Other traffic is not hit by the configured access-list and will therefore not
hit the crypto map and traffic will not be encrypted. Only the IP addresses that are
configured in the access-list will hit the crypto map and will be encrypted.
Now we need to finish the configuration for the second MDS.
MDS2
MDS2(config)# feature crypto ike
MDS2(config)# feature crypto ipsec
MDS2(config)#
MDS2(config)# crypto ike domain ipsec
MDS2(config-ike-ipsec)# identity address
MDS2(config-ike-ipsec)# key IPexpertEncrypt address 198.18.69.9
MDS2(config-ike-ipsec)# policy 1
MDS2(config-ike-ipsec-policy)# encr aes
MDS2(config-ike-ipsec-policy)# hash md5
MDS2(config-ike-ipsec-policy)# authentication pre-share
MDS2(config-ike-ipsec-policy)# group 2
MDS2(config-ike-ipsec-policy)# exit
MDS2(config-ike-ipsec)#
MDS2(config-ike-ipsec)# ip access-list FCIP1IPSEC permit ip 198.18.69.11 0.0.0.0
198.18.69.9 0.0.0.0
MDS2(config)# crypto transform-set domain ipsec FCIP1 esp-aes 128 esp-md5-hmac
MDS2(config)# crypto map domain ipsec FCIP 1
MDS2(config-crypto-map-ip)# match address FCIP1IPSEC
MDS2(config-crypto-map-ip)# set peer 198.18.69.11
MDS2(config-crypto-map-ip)# set transform-set FCIP1
MDS2(config-crypto-map-ip)# interface gig1/1.69
Now all FCIP 1 related traffic is secured and we have finished task 3!
300
Task 4: SAN Extension Tuner
The next task will let you configure a specific MDS feature that will let you generate test traffic
to tune your FCIP configurations. This feature is called the SAN Extension Tuner.
The test traffic that is generated so you are able to optimize your FCIP throughput according to
the IP network latency, jitter and throughput.
You will create a virtual initiator and a virtual target at each end of an FCIP
connection. Between this initiator and target you will create a stream of I/Os just like normal
FC traffic. By using this data you are able to optimize the FCIP connection.
There are a number of requirements and best practices to follow when you want to use this
feature.
The virtual ports do not register with the name-server
The virtual ports will not show up for other initiators and targets in the Fibre
Channel fabric.
Login requests are rejected
When a request is sent to the FCID the request is rejected
Virtual hosts can only communicate to each other
The virtual initiator and virtual target are only able to communicate to
each other and not to other hosts or targets in the Fibre Channel fabric
Use a different GigabitEthernet interface than the FCIP interface
As you are using this feature to test the ethernet interface that is used for the FCIP
connection, you should not use the same GigabitEthernet interface as the tunnel
itself because it impacts the amount of traffic in the link
This interface does not need to be up, it is just used to generate the traffic on
iSCSI needs to be enabled
Separate VSAN is recommended

The use of this feature in a separate VSAN is not strictly necessary, because all traffic
towards the virtual hosts is rejected, but its still recommended to configure this
feature in a separate VSAN.
Configuration can be extensive. You are required to use manually configured nWWNs and
pWWNs. Besides that its important to remember that you first need to configure the sanextension-tuner configuration about the virtual initiator/target on each switch
before you start configuring the commands. Otherwise the command for the read or write
action is not accepted when the virtual host is not found.
301
Be aware that you require zoning changes before the hosts are able to communicate to each
other. As we have no zoning configured for VSAN 50, which is the VSAN we should use in
this task. We can just allow traffic to be sent in this VSAN in the Default Zone.
We require to enable a couple of things before we start the san-extension-tuner
configuration.
MDS1
MDS1(config)# feature san-ext-tuner
MDS1(config)# feature iscsi
MDS1(config)# int gig1/3
MDS1(config-if)# int iscsi1/3
^
Invalid range at '^' marker.
MDS1(config)# iscsi enable module 1
MDS1(config)# int iscsi1/3
We also need to enable the iSCSI feature for this task. When you enable iSCSI you also need
to enable the iSCSI interface. The interface numbering is equal to the GigabitEthernet
number. Before this interface exists, you also need to enable the iSCSI feature on the line
card (module) that you want to enable the iSCSI interface on. In the case of a fixed chassis,
like the MDS 9222i, we only have 1 module.
After its enabled we can enable the san-extension-tuner host in the network and we need
to enable the Default Zone to allow traffic for being sent between hosts.
MDS1
MDS1(config)# exit
MDS1(config)# san-ext-tuner
^
% Incomplete command at '^' marker.
Weird enough, the san-extension-tuner configuration is not entered using the

configuration mode, but from the operational mode.
MDS1(config)# exit
MDS1# san-ext-tuner
MDS1(san-ext)# ?
nWWN
Configure san_ext_tuner nwwn
no
nport Configure nport
end
Go to exec mode
exit
pop
push
where Shows the cli context you are in
MDS1(san-ext)# nwwn ?
<hh:hh:hh:hh:hh:hh:hh:hh>
Enter the N-WWN of san extension tuner
MDS1(san-ext)# nwwn 10:00:00:00:00:00:00:01

302

MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 ?
vsan Configure nport vsan
MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 ?
interface Specify the interface
MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface ?
gigabitethernet Ethernet interface
MDS1(san-ext)# nport pwWN 10:00:00:00:00:00:00:02 vsan 50 interface gig1/3
After the N-port is configured you can see a successful FLOGI being done in the configured
VSAN with the nWWN and pWWN that you just configured.
MDS1(san-ext-nport)# sh flogi data v 50
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------iscsi1/3
50
0x6c0000 10:00:00:00:00:00:00:02 10:00:00:00:00:00:00:01
MDS1(san-ext-nport)#
Now we need to repeat these steps on the second MDS.

MDS2
MDS2(config)# feature san-ext-tuner
MDS2(config)# exit
MDS2# san-ext-tuner
MDS2(san-ext)# nwwn 20:00:00:00:00:00:00:01
Now we have 2 virtual hosts in VSAN 50 that we can use to create our data
communication flows on for our san-extension-tuner feature.
Now we configure the read flow towards 1 direction.

MDS1
MDS1# san-ext-tuner
MDS1(san-ext-nport)# read command-id 1 target 20:00:00:00:00:00:00:02 transfer-size
512000 outstanding-ios 2 ?
<CR>
continuous
Do transactions forever
303

num-transactions Configure num transactions
512000 outstanding-ios 2 continuous
Do not forget to enable the continuous keyword at the end of the configuration. The
question clearly states that you should start a continuous flow of traffic
We currently have traffic in one direction. The task states that we should enable the read
commands in both directions. So we need to configure the same configuration on the other
MDS switch as well.
MDS2
MDS2# san-ext-tuner
512000 outstanding-ios 2 continuous
After this configuration is applied the read command immediately starts and traffic is flowing
across the FCIP 2 connection, because this is the only FCIP connection where VSAN 50 is
allowed.
Task 5: iSCSI
The next task is all about the iSCSI feature on the Cisco MDS platform. This feature has been
available since a long time and is still available, however a more newer version is also available,
which is discussed in the next task.
Using the iSCSI feature on the MDS platform you are able to extend IP connected hosts
(initiators) on to the Fibre Channel network. This means that the Fibre Channel fabric will have
an entry in the name-server for this IP host just like if it was a Fibre Channel connected
server.
304
On the other side, you can create iSCSI targets based on Fibre Channel targets. You can
take any target in the Fibre Channel fabric and make this accessible through iSCSI.
The iSCSI feature is available on the same IP interfaces as the FCIP feature is available.
The MDS switch makes the conversion between the IP traffic and Fibre Channel traffic. Hosts
that log-in using iSCSI do not know they are actually communicating with a Fibre Channel
switch and a Fibre Channel disk. The iSCSI header is stripped and a Fibre Channel header is
placed on top of it. Of course the return traffic works exactly vice versa.
The drawing below illustrates the working of the iSCSI portal on the MDS switch.
The most easy option for configuring iSCSI is to use the dynamic mapping feature.
With this feature its possible to present every target available on the Fibre Channel network as
an iSCSI target. As the first question states in this task this is prohibited!
This leaves you with the option to configure static targets. Here you have options to
configure targets either on a target basis or you can even specify which LUN can be accesses
through the iSCSI network.
As with targets you can also dynamically create virtual initiators. Meaning that every
device that tries a log-in on the MDS switch IP address on the iSCSI portal, will
automatically get a virtual initiator assigned. Again we do not want this and we will be
configuring static initiators so we have full control over which host can access which
target, as the initiators will need to be included in zoning!
The 2 options that you can use to define a iSCSI initiator are:
iSCSI Qualified Name (iQN)
This IQN is a clear-text name that the iSCSI device uses to identify itself. When an
initiator has multiple IP addresses it will still be just a single Virtual N-Port in the
Fibre Channel fabric
305
IP address
The Virtual N-port can also be created according to the IP address that the iSCSI
initiator uses to access the MDS. Each IP address is another virtual N-port and
another host in the Fibre Channel fabric
In the initial configuration we will be using the Transparent Initiator mode, which
means that every iSCSI initiator will appear as an Initiator in the FC fabric network
(Virtual N-port).
During this task we will also be configuring the other mode, which is the Proxy Initiator
mode. With the use of this mode. The MDS switch will act as a single virtual N-port and all
iSCSI initiators are hidden behind this proxy feature. This means that you can not use
zoning to determine which iSCSI initiator can access which target.
However there are multiple ways in the iSCSI configuration where you can specify which
target can access which initiator.
Now we start the configuration of our first couple questions in this iSCSI task.
We call an iSCSI interface on the MDS switch a iSCSI Portal, as its a single portal where
you can log-in and access all targets that you have the right to.
We start with some infrastructure configuration to create the iSCSI portal.
The iSCSI interface that will be configured here is a virtual interface that is an overlay on
top of the GigabitEthernet interface. The numbering of the iSCSI interface is equal to the
GigabitEthernet numbering. This interface is required to be enabled, to really activate the
iSCSI portal on the MDS. Besides that, which we already did in the previous task, is that we
need to enable the iSCSI feature globally and per line-card where the feature needs to be
enabled.
We configure the VLAN according to Drawing 2. There we find that we should be using VLAN
71. Therefore we need to create another sub-interface for the VLAN. Its not possible on the
Cisco MDS switch to configure the iSCSI portal on a sub-interface. Meaning that all iSCSI
interface configuration that you do will be used on all sub-interfaces that are configured.
This means that you can not create different iSCSI portals on different sub-interfaces, but
only a single portal per physical interface.
MDS1
MDS1(config-if)# int gi1/1.70
306

MDS1(config-if)# int iscsi1/1.70
^
Invalid interface format at '^' marker
Like previously mentioned. You can not create an iSCSI sub-interface only a global interface.
MDS1(config-if)# port 3261
MDS1(config-if)# tcp ?
cwm
keepalive-timeout
max-bandwidth-kbps
max-bandwidth-mbps
Configure maximum bandwidth
max-jitter
Configure jitter for iSCSI in micro seconds
pmtu-enable
sack-enable
send-buffer-size
The TCP parameters that can be configured are similar to the options that are available for
configuring FCIP connections.
MDS1(config-if)# qos ?
<0-63> QOS code point value
MDS1(config-if)# qos 22
MDS1(config-if)# vsan data
MDS1(config-vsan-db)# vsan 10 interface iscsi1/1
Finally the last configuration that needs to be done is by putting the iSCSI interface into
another VSAN, because the Default VSAN is suspended. When this is not configured, no
matter what kind of VSAN configuration is done under the virtual initiators or virtual
targets the interface will remain down, because the Default VSAN is suspended.
The different topics in this infrastructure task is that the portal should use a non-default
port. By default iSCSI uses TCP 3260 for all communications. We now change this to TCP
3261. The configuration is done under the iSCSI interface. Under the iSCSI interface the
characteristics of the traffic, like TCP parameters, port number, etc.
MDS1(config-if)# do sh int iscsi1/1
iscsi1/1 is up
Port WWN is 20:13:00:0d:ec:54:a9:80
Admin port mode is ISCSI
Port mode is ISCSI
Port vsan is 10
Speed is 1 Gbps
Interface last changed at Tue Oct 23 14:13:46 2012
iSCSI initiator is identified by name
Number of iSCSI session: 0 (discovery session: 0)
Number of TCP connection: 0
Configured TCP parameters
Local Port is 3261
307

PMTU discover is enabled, reset timeout is 3600 sec
Keepalive-timeout is 60 sec
Minimum-retransmit-time is 300 ms
Max-retransmissions 4
Sack is enabled
QOS code point is 22
Configured maximum jitter is 500 us
Forwarding mode: store-and-forward
TMF Queuing Mode: enabled
Proxy Initiator Mode: disabled
iSCSI statistics
Input 0 packets, 0 bytes
Command 0 pdus, Data-out 0 pdus, 0 bytes, Unsolicited 0 bytes
Output 0 packets, 0 bytes
Response 0 pdus (with sense 0), R2T 0 pdus
Data-in 0 pdus, 0 bytes
Now our iSCSI portal is up and running, but since we are using no dynamic targets or
initiators nothing will be able to log-in to it yet.
As shown in this output, you can see that by default initiators are recognized by their name
(iQN). This needs to be changed for this portal, as we want to use the IP address of the
initiator as we look at the following question. This can be changed per iSCSI interface and
therefore per iSCSI portal configured on the MDS switch.
MDS1
MDS1(config-if)# switchport ?
description
Enter description of maximum 80 characters
initiator
Configure iSCSI initiator id mode
proxy-initiator Configure iSCSI proxy initiator mode
timestamp-check Enable timestamp check
MDS1(config-if)# switchport initiator ?
id Configure iSCSI initiator id mode
MDS1(config-if)# switchport initiator id ?
ip-address Configure iSCSI initiator id mode based on ip-address
name
Configure iSCSI initiator id mode based on init name
MDS1(config-if)# switchport initiator id ip-address
308
The Cisco MDS switch allows you to configure both manual nWWN and pWWNs to the initiators
and you can chose for the feature that the MDS will select pWWNs or a nWWN automatically
based on a pool.
In this case we will use statically configured WWNs.
MDS1
MDS1(config)# iscsi initiator ip-address 198.18.71.100
MDS1(config-iscsi-init)# ?
mutual-chap Assign username for initiator's challenge (mutual chap)
no
static
Create static nWWN or pWWN for the initiator
username
Assign username for iSCSI login authentication
vsan
Assign VSAN membership for the initiator
end
Go to exec mode
exit
pop
push
where
MDS1(config-iscsi-init)# static nWWN 11:00:00:00:00:00:00:01
MDS1(config-iscsi-init)# static pwwn ?
<hh:hh:hh:hh:hh:hh:hh:hh> Enter pWWN
system-assign
Ask the system to assign static pWWN(s) for this
initiator
MDS1(config-iscsi-init)# static pwwn 22:00:00:00:00:00:00:01
We need to ensure that the initiator is participating in VSAN 20, but we can not configure
this under the initiator by using the vsan command in this case. With the use of this command
you are able to configure different initiators in different VSANs. This creates a lot of
flexibility
in
the
deployment
of
the
initiators.
We just configured the iSCSI interface under the VSAN database. So we need to change this to
VSAN 20 to support our question and let the initiator have access to resources in VSAN 20.
MDS1
MDS1(config-if)# vsan data
309
Now that we have finished our configuration of our initiator we can verify its configuration.
Dont get scared when you issue the first command and you dont see any results.
This has to do with the fact that the show iscsi initiator command will only show you
the currently active iSCSI sessions. With the configured keyword behind it, it will show you
the configuration of your iSCSI initiators.
You will also not see any FLOGIs or Name-Server entries until the iSCSI session is active
and working.
MDS1
MDS1(config-vsan-db)# sh iscsi initiator
MDS1(config-vsan-db)# sh iscsi initiator ?
*** No matching command found in current mode, matching in (exec) mode ***
<CR>
>
Redirect it to a file
>>
Redirect it to a file in append mode
WORD
Enter initiator node name (Max Size 223)
configured
Show iSCSI initiator configured information
detail
Show more detailed information
fcp-session
Show FC session details
iscsi-session Show iSCSI session details
summary
Show iSCSI initiator summary information
|
Pipe command output to filter
MDS1(config-vsan-db)# sh iscsi initiator
iSCSI Node name is 198.18.71.100
configured
Node WWN is 11:00:00:00:00:00:00:01

No. of PWWN: 1
Port WWN is 22:00:00:00:00:00:00:01
Configured node (iSCSI)
310
Now that the initiator part is configured we can start with configuring targets for this
iSCSI host. The host should be able to access the JBOD 2 Disk 1 in VSAN 20.
The next question states that you should be configuring a virtual target for the JBOD
Disk 1. This is done using a similar configuration as the initiator.
Before the virtual target can be accessed you need to configure which initiators are
allowed to access it. This means, the initiators that will see it when they discover the
targets through iSCSI. After this configuration you still need zoning where you need to give
the virtual pWWN or IP-address access to the Fibre Channel target.
Next is that you can configure the virtual-target to have it showing up on which iSCSI
portal. This is done by specifying the GigabitEthernet interface (and with that the
portal) where its available.
Now on to the configuration.
MDS1
MDS1(config)# iscsi virtual-target name iqn.iscsi-disk-JBOD2-Disk1
MDS1(config-iscsi-tgt)# ?
advertise
Advertise virtual-target on interfaces specified
all-initiator-permit Allow all iSCSI initiator access to this target
initiator
Allow iSCSI initiator access to this target
no
pWWN
Enter the pWWN of the fc-target
revert-primary-port
Revert back to primary port when it comes back up
end
Go to exec mode
exit
pop
push
where
MDS1(config-iscsi-tgt)# initiator ?
WORD Enter iSCSI initiator name (Max Size 223)
ip
Allow iSCSI initiator access to this target by ip address
MDS1(config-iscsi-tgt)# initiator ip address 198.18.71.100 ?
A.B.C.D Enter initiator ip netmask
permit
Allow iSCSI initiator access to this target
MDS1(config-iscsi-tgt)# initiator ip address 198.18.71.100 permit
You can specify an entire range of IP addresses with this command, when you do not configure
any netmask, it means that you only allow a single host.
MDS1
MDS1(config-iscsi-tgt)# advertise ?
interface Interface eth-interface-to-advertise
MDS1(config-iscsi-tgt)# advertise int gig1/1.71
MDS1(config-iscsi-tgt)# pwwn ?
311
Within the current iSCSI configuration its only possible to configure a pWWN of the target you
want to make available. You will see in the next task that there are more options available, like
using the device-alias instead of the pWWN, which is a much more flexible way.
MDS1
MDS1(config-iscsi-tgt)# pwwn 21:00:00:11:0d:18:fe:ce ?
<CR>
fc-lun
FC lun
secondary-pWWN Enter the secondary pWWN of the fc-target
The other options are not used in the current question.

MDS1
MDS1(config-iscsi-tgt)# pwwn 21:00:00:11:0d:18:fe:ce
MDS1(config-iscsi-tgt)# exit
To finish up, we need to configure zoning so the host as access to the target. This can be
based on multiple things.
MDS1
MDS1(config)# zone name iSCSI_1 vsan 20
MDS1(config-zone)# member ?
device-alias
Add device-alias member to zone
domain-id
Add member based on domain-id,port-number
fcalias
Add fcalias to zone
fcid
Add FCID member to zone
fwwn
Add Fabric Port WWN member to zone
interface
Add member based on interface
ip-address
Add IP address member to zone
pwwn
Add Port WWN member to zone
symbolic-nodename Add Symbolic Node Name member to zone
You can use the following options in letting iSCSI initiators access Fibre Channel
targets:
IP address
Just use the IP address of the initiator, just like we configured the virtual
initiator
IQN
Use the hostname of the initiator, called the symbolic-nodename in the

zoning configuration to configure the initiator in the zone. This is a flexible way
independent
of
the
IP
addressing
of
the
initiator
312
pWWN
You can use the virtual pWWN that is manually assigned by you or system assigned by
the MDS switch to give the initiator access to the FC Disk. This is also a very flexible
option, but it makes you zoning configuration harder to read
MDS1
MDS1(config-zone)# member ip-address 198.18.71.100
MDS1(config-zone)# zoneset name ZS1 vsan 20
MDS1(config-zoneset)# member iSCSI_1
MDS1(config-zoneset)# zoneset activate name ZS1 vsan 20
MDS1(config)#
The next question requires you to configure authentication. Its possible to authenticate your
iSCSI session in both directions, which is what we are looking for.
You can authenticate the iSCSI users according to the local database, which is what we are
going to use here, or through an RADIUS or TACACS+ server.
Be aware that when you are configuring the user, to not forget the special iscsi keyword
behind the line, otherwise it will definitely not work!
Especially when you are configuring mutual-chap, where the second authentication is
configured right under the virtual initiator, you might think that it enough, but you are
not done there. You need to configure the initial CHAP username/password as well!
MDS1
MDS1(config)# iscsi initiator ip-address 198.18.71.100
MDS1(config-iscsi-init)# username iSCSI1
MDS1(config-iscsi-init)# mutual-chap ?
username Assign username for initiator's challenge (mutual chap)
MDS1(config-iscsi-init)# mutual-chap username iSCSI1 ?
password Assign password for initiator's challenge (mutual chap)
MDS1(config-iscsi-init)# mutual-chap username iSCSI1 password IP3xp3rtiSCSI
MDS1(config-iscsi-init)# exit
MDS1(config)# username iSCSI1 password IP3xp3rtiSCSI iscsi
After this the configuration of this virtual target is done and we can verify our
configuration.
MDS1(config)# show iscsi virtual-target
target: iqn.iscsi-disk-jbod2-disk1
Port WWN 21:00:00:11:0d:18:fe:ce
No. of advertised interface: 1
GigabitEthernet 1/1.71
No. of initiators permitted: 1
initiator 198.18.71.100/32 is permitted
All initiator permit is disabled
Trespass support is disabled
313

Revert to primary support is
disabled
MDS1(config)#
All configuration parameters that we configured are available. Now we check our changed
initiator, where we added the authentication.
MDS1(config)# show iscsi initiator configured
iSCSI Node name is 198.18.71.100
Node WWN is 11:00:00:00:00:00:00:01
No. of PWWN: 1
Port WWN is 22:00:00:00:00:00:00:01
User Name for login authentication: iSCSI1
User Name for Mutual CHAP: iSCSI1
MDS1(config)#
The authentication is added and we see that the same username is used for both the login and
the second CHAP (mutual).
The next question asks you to configure another virtual target with a LUN specified. The
iSCSI feature supports LUN masking, so you are able to convert the LUN IDs of targets to
another LUN ID on the iSCSI level.
The question will also be using another great feature of the iSCSI feature of the Cisco MDS
platform. The iSCSI virtual-target feature can take care of high availability. Where it will
be using a sort of SDV (SAN Device Virtualization) technology to ensure this. The
iSCSI virtual target will have access to a primary pWWN and a secondary pWWN that it can
use to failover to as soon as the primary fails.
Configuration can generate quite a long command!
MDS1
MDS1(config)# iscsi virtual-target name iqn.iscsi-disk-jbod1-disk3
MDS1(config-iscsi-tgt)# sh device-alias data
MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa ?
iscsi-lun ISCSI lun
MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa iscsi-lun 0x0 ?
<CR>
secondary-pWWN Enter the secondary pWWN of the fc-target
MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa iscsi-lun 0x0
secondary-pWWN 21:00:00:11:0d:18:a0:ce ?
<CR>
sec-lun FC lun on the secondary port
314

MDS1(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fc:ce fc-lun 0xa iscsi-lun 0x0
secondary-pWWN 21:00:00:11:0d:18:a0:ce sec-lun 0xa
Here we can see that we can even configure a different Fibre Channel LUN for both the primary
and the backup target. As the question only specified LUN 10, we will continue to use this on
the backup target as well.
MDS1
MDS1(config-iscsi-tgt)# all-initiator-permit
As the question stated that iSCSI initiators should have access to the virtualtarget, this indicates that we want to allow all initiators that will be logging in, to have
access to this specific target. As we only have 1 initiator configured, this will be currently
the only one that has access to it.
MDS1
MDS1(config-iscsi-tgt)# revert-primary-port
The question also stated that the backup disk should fall back to the primary when this is
available again after repair.
MDS1
MDS1(config-iscsi-tgt)# trespass
MDS1(config-iscsi-tgt)#
The trespass support that is asked for, is currently a hidden command, but still available in
the configuration and a supported feature. This feature ensures that Logical Units are
transferred over to the failover target.
Do not forget to enable zoning for the initiator that we configured! In this zoning its
very important not to forget to add both the disks that you are using, so both the primary and
the backup target should be mentioned in the zone, before the initiator has access to it.
MDS1
MDS1(config)# zone name iSCSI_2 vsan 20
MDS1(config-zone)# member ip-address 198.18.71.100
MDS1(config-zoneset)# member iSCSI_2
MDS1(config)#
Now we can verify our new target.

MDS1(config-iscsi-tgt)# show iscsi virtual-target
Port WWN 21:00:00:11:0d:18:fe:ce
No. of advertised interface: 1
GigabitEthernet 1/1.71
315

initiator 198.18.71.100/32 is permitted
Trespass support is enabled
Revert to primary support is disabled
Port WWN c6:83:00:11:0d:18:fc:ce
Secondary PWWN 21:00:00:11:0d:18:a0:ce
No. of LU mapping: 1
iSCSI LUN: 0x0000, FC LUN: 0x000a, Secondary FC LUN: 0x000a
All initiator permit is enabled
Trespass support is enabled
Revert to primary support is enabled
In the second virtual target you will find the 2 pWWNs of the disks and the features that are all
enabled.
The final feature that should be enabled on this iSCSI portal is that we want you to improve
the read performance on the switch.
This is related to the way the iSCSI packets are treated in the MDS switch. The MDS can treat
iSCSI frames just like an Ethernet switch has different modes to transfer Ethernet packets.
The modes that are available are:
Pass-Thru
The downside of this mode is that it lowers the total amount of data transfer that is
possible. However the positive side is that you get lower latency and you are able to use
the data-digest feature.
Store-and-Forward
This is the default and recommended mode. Data-digest can not be used, but you
do get a faster data transfer performance in the iSCSI process. The entire iSCSI
packet is cached inside the box before the next action is taken.
Cut-Thru
This mode improves the read performance when compared to store-and-forwards.

The downside is that the improvement only works for read performance and for initial
packets only.
The changing of the mode is also a hidden command, but you are still able to configure all three
modes and they are supported. In this case we want to improve read performance, so we
should be using the Cut-Thru switching mode.
MDS1
316

MDS1(config-if)# mode ?
store-and-forward Enable store-and-forward
MDS1(config-if)# mode cut-thru
The different modes are not available through the CLI context help, but they are available. You
can verify using the following output.
MDS1(config-if)# do sh int iscsi1/1
iscsi1/1 is down (Initializing)
Port WWN is 20:13:00:05:9b:70:5a:00
Port vsan is 20
iSCSI initiator is identified by ip address
Local Port is 3261
Sack is enabled
QOS code point is 0
Forwarding mode: cut-thru
Proxy Initiator Mode: disabled
iSCSI statistics
MDS1(config-if)#
Next we are configuring a new iSCSI portal on the other MDS switch.
This new portal is configured in the same iSCSI VLAN as we used in the previous questions,
but as we are using another IP address on another MDS switch, there is no problem in any
overlap.
This portal will work a little different than the previous one as the iSCSI initiators should
all be hidden behind a single nWWN and pWWN. This is called the proxy-initiator-mode.
With this mode the MDS switch will act as the Fibre Channel host in the network where all
traffic is sent and the switch will convert it then to iSCSI packets destined to the correct
initiator.
317
This mode introduces flexibility as you only have a single entry in your FC Name-Server for the
iSCSI hosts. It hides the iSCSI hosts from the network, but it eliminates the fact that you can
zone the targets and initiators together. It also creates simplicity as you have most
security features to secure the combination of initiators and targets in the virtual-target
configuration as you would have in zoning. Therefore the adding of each initiator and
each target (including possible backup targets) will create a lot of additional zones that
need to be configured.
Within the proxy-initiator-mode, the MDS will create a virtual N-port per iSCSI
portal (per iSCSI interface to be exact). So when you enable this mode on multiple
iSCSI interfaces, it will create additional N-ports in the FC Fabric, which you all need to
add to the zones.
The following drawing illustrates how the Proxy Initiator Mode works.
We now configure the iSCSI portal on MDS2. Most configuration is equal to the other iSCSI
portal on MDS1, except that the behavior is very different.
Keep in mind that when you configure proxy-initiator-mode, its not possible anymore
to do your zoning based on IQNs or IP addresses.
MDS2
MDS2(config)# iscsi enable mod 1
MDS2(config)# int gig1/1.71
MDS2(config-if)# int gig1/1
318
We are first required to enable iSCSI on this switch as it wasnt enabled before.
MDS2
MDS2(config-if)# switchport proxy-initiator ?
<CR>
nWWN
Configure nWWN for proxy initiator mode
MDS2(config-if)# switchport proxy-initiator nwwn ?
<hh:hh:hh:hh:hh:hh:hh:hh> Enter nWWN
MDS2(config-if)# switchport proxy-initiator nwwn 11:22:33:44:55:66:77:88 ?
pwwn Configure pwwn for proxy initiator mode
MDS2(config-if)# switchport proxy-initiator nwwn 11:22:33:44:55:66:77:88 pwwn ?
MDS2(config-if)# switchport proxy-initiator nwwn 11:22:33:44:55:66:77:88 pwwn
aa:bb:cc:dd:ee:ff:00:11
MDS2(config-if)#
In this example we are using manually configured nWWN and pWWN for the proxy host that is
generated on the switch. This makes it easier to define our zoning later on as we do not need
to look up the values, but we configured them by ourselves. Be aware that this should be a
globally unique value. In the CCIE Data Center lab exam you do not have that many Fibre
Channel connected devices, so it should be fine to use these kinds of values for lab purposes.
Now we can verify our proxy-initiator mode configuration.
MDS2(config-if)# sh int iscsi1/1
iscsi1/1 is up
Port WWN is 20:13:00:05:9b:77:0d:c0
Port mode is ISCSI
Port vsan is 1
Speed is 1 Gbps
iSCSI initiator is identified by name
Local Port is 3260
Sack is enabled
QOS code point is 0
Proxy Initiator Mode: enabled
319

nWWN is 11:22:33:44:55:66:77:88 (manually-configured)
pWWN is aa:bb:cc:dd:ee:ff:00:11 (manually-configured)
iSCSI statistics
The next question requires you to configure the data-digest feature. The funny thing is
that this is a feature that can not be directly configured. However, as just discussed, this feature
is
only
available
in
a
certain
forwarding
mode.
When enabling this feature the data-digest feature is automatically enabled when the
iSCSI initiators send requests for this!
MDS2
MDS2(config-if)# mode pass-thru
Now we are going to configure 3 initiators that will be logging in to the iSCSI portal.
They are member of VSAN 10, but we cant configure the VSAN database for this VSAN.
Remember that the iSCSI portal is currently in the suspended Default VSAN and will
not come online!
MDS2
MDS2(config)# iscsi initiator name iqn.initiator-server-1
MDS2(config-iscsi-init)# vsan 10
MDS2(config-iscsi-init)# iscsi initiator name iqn.initiator-server-2
MDS2(config-iscsi-init)# iscsi initiator name iqn.initiator-server-3
We can configure initiators to be in any VSAN. This also means that the same iSCSI portal
can be using resources from different VSANs. In each VSAN the switch will create a virtual
N-port when necessary.
MDS2
MDS2(config-iscsi-init)# vsan data
320
To overcome the limitation that we can not configure VSAN 10 in the VSAN database
configuration. We just enable the iSCSI portal interface in another VSAN. This has no impact
on the working of the initiators. As the configuration under the initiator configuration
is leading over the VSAN database configuration.
Next we need to give the initiators access to a virtual target.
MDS2
MDS2(config-iscsi-tgt)# sh device-alias database
MDS2(config-iscsi-tgt)# iscsi virtual-target name iqn.jbod-disk-J1D1
MDS2(config-iscsi-tgt)# pwwn c6:83:00:11:0d:18:fa:ce
MDS2(config-iscsi-tgt)# initiator ?
WORD Enter iSCSI initiator name (Max Size 223)
ip
Allow iSCSI initiator access to this target by ip address
MDS2(config-iscsi-tgt)# initiator iqn.initiator-server-1 permit
After configuring the virtual-target we can verify our target and initiator
configuration.
MDS2(config-iscsi-tgt)# sh iscsi virtual-target
target: iqn.jbod-disk-j1d1
Port WWN c6:83:00:11:0d:18:fa:ce
initiator iqn.initiator-server-1 is permitted
We can see that the target only allows certain initiators to connect to it and that it is
connected to the pWWN of J1D1.
MDS2(config-iscsi-tgt)# show iscsi initiator configured
iSCSI Node name is iqn.initiator-server-1
Member of vsans: 10
321

Member of vsans: 10
Member of vsans: 10
The initiators are configured so that they are members of VSAN 10.
Finally we need to enable zoning for these initiators and a target. Where we can only
use 2 entries in the zoning.
This is relatively easy as we are using the proxy initiator mode, so we can use the pWWN
of the proxy host and the device-alias of the J1D1 disk and we have our zoning
completed.
MDS2(config-iscsi-tgt)# zone name ISCSI1 vsan 20
MDS2(config-zone)# zone name ISCSI1 vsan 10
MDS2(config-zone)# member pwwn aa:bb:cc:dd:ee:ff:00:11
MDS2(config-zoneset)# member ISCSI1
MDS2(config)#
322
With this configuration we have our iSCSI configuration completed. In the next task we will be
configuring the next version of the iSCSI feature on the MDS platform called iSLB!
Task 6: iSLB
This final task of this chapter will be about configuring iSLB. This feature is the next version of
iSCSI with more features and has more features specifically related to high availability and
load balancing between different MDS switches.
The iSLB feature has a couple unique features that are missing from the iSCSI
implementation, which might make it hard to roll-out this feature for hundreds of iSCSI
initiators.
The scaling is accomplished by the following features:
Zoning can be automatically done with auto-zoning
By the use of auto-zoning you are able to automatically zone the initiator and
virtual-target together without having to manually configure the zoning. The
combination of the initiator and virtual-target is done by the use of so-called
initiator targets. This means that under the initiator configuration the
virtual-target configuration is done.
Distribution using CFS is possible
By using CFS distribution for this feature you can push the configuration of the
iSLB feature to multiple MDS switches in the fabric. This eases the configuration and
ensures that you have equal configurations for initiators and targets across the
whole fabric.
Load balancing of initiator requests across MDS switches using VRRP and iSCSI redirect
When you are using VRRP, there is a single switch that will receive all iSCSI login
requests. The iSLB feature implements iSCSI login redirection so the login request is
redirected to the other MDS switch which is also participating in the VRRP configuration.
323
Keep in mind that you need to keep iSCSI enabled on the switch as well. This is required when
you are using more than 200 iSLB initiators, but its always a best practice to follow.
iSLB can be using dynamic initiators and targets just like iSCSI. By default the auto
learning is using the iSCSI feature, but you can change this to use the iSLB feature instead.
When the initiators or targets are already learned through the iSCSI feature you are
not able to convert these to iSLB variants. You need to remove them and re-discover the
initiators and targets.
Again in our task we are not allowed to use any dynamic learning, just like for iSCSI.
The initial questions in this task are related to building up the iSLB portal on a different
interface than were we configured the iSCSI portal.
When we read the first 3 questions we have enough information to start building our portal.
We can only configure MDS2 for configuration changes to initiators and targets. Which
means that we should be using distribution of configuration using the CFS protocol.
The third question indicates that the portal should be high available. Where the only option
we have is to use the VRRP.
Lets start the configuration on our switches:
MDS1
MDS1(config)# iscsi enable
MDS1(config-if-vrrp)# prio 110
MDS1(config-if-vrrp)# int iscsi1/2
Traffic on iscsi1/2 may be impacted. Do you want to continue? (y/n) [n] y
We first enable the feature and configure the VRRP group. We make MDS1 the primary switch.
Which switch is the primary, is not given in this case in the question, so we can take any switch
we want for this use. Do not forget to put the iSCSI interface into another VSAN, because
VSAN 1 is still suspended and interface in that VSAN will not come online unless they are
changed
to
another
VSAN
which
is
unsuspended.
324
The question about the configuration that can only be done on MDS2, is about enabling the
distribution of the iSLB feature using the CFS protocol. This should be enabled.
MDS1
MDS1(config)# islb distribute
Now we can verify our configuration.

MDS1(config-if)# sh vrrp
Time Pre State
VR IP addr
--------------------------------------------------------------------------GigE1/2.72 72
IPv4
110
1 s
master 198.18.72.1
MDS1(config-if)# sh islb ?
*** No matching command found in current mode, matching in (exec) mode ***
cfs-session
Show iSLB cfs session information
initiator
Show iSLB initiator information
merge
Show iSLB merge information
pending
Show iSLB pending configuration
pending-diff
Show iSLB pending configuration diffs
session
Show iSLB session information
status
Show iSLB CFS status
virtual-target Show iSLB virtual-targets
vrrp
Show iSLB vrrp load balance information
MDS1(config-if)# sh islb status
iSLB Distribute: Enabled
iSLB CFS Session: Does not exist
Number of load balanced VRRP groups: 0
Number of load-balanced initiators: 0
MDS1(config-if)#
Now MDS2 should be configured for the same features.

MDS2
MDS2(config)# iscsi enable
MDS2(config-if-vrrp)# int iscsi1/2
MDS2(config-if)# islb distribute
Traffic on iscsi1/2 may be impacted. Do you want to continue? (y/n) [n] y
325
The next step in our task is that we need to enable the load balancing feature on the iSLB
activated switches. The load balancing feature only works in conjunction with the VRRP. It is
based on a weight that is given to an initiator. The load balancing is always using the
number of initiators to load balance, not to the number of sessions on the switch.
You can specify a weight to an initiator to take care of the load balancing. When a host has
a lower bandwidth connection, it can be assigned a lower weight, as the amount of load it
generates to the MDS switch is lower than other servers. So the MDS can take more of these
smaller bandwidth servers on one interface.
Another use case for using load balancing is when there are certain initiators that you know are
going to set-up a lot of iSCSI sessions. Therefore more load is generated on the MDS and the
weight number should take care of that.
The default value is 1000.
Load balancing is enabled with only a single command.
MDS1
MDS1(config)# islb ?
abort
Abort pending iSLB configuration
commit
Commit pending iSLB configuration
distribute
Enable CFS for iSLB configuration
initiator
Configure iSLB initiator
save-initiator Make WWNs for iSLB initiator persistent
virtual-target Configure iSLB virtual-target
vrrp
Configure iSLB on this vrrp group
zoneset
Configure iSCSI zoneset activate
MDS1(config)# islb vrrp ?
<1-255> IPv4 VR group number
ipv6
Configure vrrp IPv6 on this interface
MDS1(config)# islb vrrp 72 ?
load-balance Enable load-balancing on this vrrp group
MDS1(config)# islb vrrp 72 load-balance
MDS1(config)# islb commit
The load balancing can be monitored using the following output. All load balancing
configuration is distributed using the CFS Protocol, so both switches are aware of all changes
going on in the load balancing mechanism. This ensure that when the master fails, which keeps
track of the iSCSI login redirects, the backup switch will have all information regarding
the load balancing and is able to successfully take over the working of the switches. Another
redirect is sent to the iSCSI initiators and sessions are automatically re-initiated.
MDS1(config)# sh islb vrrp summ
-- Groups For Load Balance --------------------------------------------------------------------------------VR Id
VRRP Address Type
Configured Status
326

-------------------------------------------------------------------------------72
IPv4
Enabled
-- Interfaces For Load Balance --------------------------------------------------------------------------------Initiator Redirect
VR Id
VRRP IP
Switch WWN
Interface
Load
Enabled
-------------------------------------------------------------------------------M
72
198.18.72.1 20:00:00:0d:ec:54:a9:80
GigE1/2.72
0
Yes
72
198.18.72.1 20:00:00:0d:ec:58:b1:aa
GigE1/2.72
0
Yes
-- Initiator To Interface Assignment --------------------------------------------------------------------------------Initiator VR Id
VRRP IP
Switch WWN
Interface
-------------------------------------------------------------------------------MDS1(config)# sh islb vrrp interface
-- Interfaces For Load Balance -Interface GigabitEthernet1/2.72
Switch wwn: 20:00:00:0d:ec:54:a9:80
VRRP group id: 72, VRRP IP address: 198.18.72.1
Interface VRRP state: master
Interface load: 0
Interface redirection: enabled
Group redirection: enabled
Number of physical IP address: 1
(1) 198.18.72.9
Port vsan: 10
Proxy initiator mode: disabled
iSCSI authentication: CHAP or None
Interface GigabitEthernet1/2.72
Switch wwn: 20:00:00:0d:ec:58:b1:aa
VRRP group id: 72, VRRP IP address: 198.18.72.1
Interface VRRP state: master
Interface load: 0
Interface redirection: enabled
Group redirection: enabled
Number of physical IP address: 1
(1) 198.18.72.11
Port vsan: 10
Proxy initiator mode: disabled
iSCSI authentication: CHAP or None
MDS1(config)#
The master switch (VRRP master) will always redirect the first request towards the backup
switch. This is because the master switch always has more load, because it needs to handle all
iSCSI login requests. Therefore it automatically gives itself an invisible weight of 1000. This
means that when 2 requests with a weight of 1000 come in to the switch it will redirect both
to the backup switch. As the master is always less busy than the backup switches. This means
that in case the load of the switches is equal, the request is always redirected to the backup.
327
The next task is only a key thing to remember that with the following questions there are no
manual zoning changes allowed. This means that we should be using the auto-zoning
feature instead. This also means that we should be using initiator-targets, as this is the
only place where the auto-zoning will work. When you only configure an initiator and a
virtual-target, these will not be auto-zoned together, because the switch does not
know they belong to each other. Therefore we will be using the initiator target feature.
We will first configure as the next question states, the 5 initiators. Remember that we are
only allowed to configure MDS2 for initiators and targets.
MDS2
MDS2(config)# islb initiator name iqn.islb-initiator-host-1
MDS2(config-islb-init)# ?
metric
Assign load balancing metric for the initiator
no
static
target
Configure iSLB initiator target
username
Assign username for iSLB login authentication
vsan
zonename
Assign zone name for the initiator targets
end
Go to exec mode
exit
pop
push
where
MDS2(config-islb-init)# static nwwn system-assign
MDS2(config-islb-init)# static pwwn system-assign 2
We need to allocate system assigned nWWNs and 2 pWWNs to the initiators. Just like we are
able when configuring iSCSI initiators.
MDS2
MDS2(config-islb-init)# zonename ?
WORD Enter zone name (Max Size 55)
MDS2(config-islb-init)# zonename islb-ipexpert-host-1
Configuring iSLB node iqn.islb-initiator-host-1 fails: Illegal character present in
the name (0x40200024)
MDS2(config-islb-init)# zonename IPexpertHost1
MDS2(config-islb-init)# islb commit
As the next question dictates we need to use zone names. By default the switch will
generate either a long hash or a name with iSLB in it. You are able to make it more readable,
by configuring the zone name under the initiator. You are not allowed to use any dashes in
the zone names. Only underscores are allowed as special character.
After the commit has been initiated, the initiator is shown in the configuration. There you
can see that the MDS allocated a nWWN and 2 pWWNs to this initiator.
MDS2(config)# sh run | section "islb initiator"
328

islb initiator name iqn.islb-initiator-host-1
static nWWN 21:1b:00:0d:ec:54:a9:82
static pWWN 21:1c:00:0d:ec:54:a9:82
static pWWN 21:1d:00:0d:ec:54:a9:82
zonename IPexpertHost1
MDS2(config)# sh islb initiator configured
iSCSI Node name is iqn.islb-initiator-host-1
Node WWN is 21:1b:00:0d:ec:54:a9:82
No. of PWWN: 2
Port WWN is 21:1c:00:0d:ec:54:a9:82
Port WWN is 21:1d:00:0d:ec:54:a9:82
Configured node (iSLB)
Zone Name for initiator targets: IPexpertHost1
Load Balance Metric: 1000
Number of Initiator Targets: 0
MDS2(config)#
The other hosts can be configured using the same commands.

MDS2
MDS2(config-islb-init)# islb initiator name iqn.islb-initiator-host-3
MDS2(config)#
The next question in our task is that Host 3 is a database server with a lot of iSCSI
connections. As we discussed earlier. The load balancing that iSLB does is based on the
number of initiators that make a connection with the switch. This particular initiator
will have a lot of iSCSI sessions with the switch, meaning that it will generate a lot more load
on the switch compared to other initiators.
We do not know how many sessions are set-up compared to other initiators, but we just
need to ensure that the load balancing takes care of this heavier initiator. Therefore we will
be configuring a little higher load.
MDS2
MDS2(config-islb-init)# metric 2000
329
Now we can verify our initiator configuration and see that everything configured has been
properly pushed to all switches in the fabric.
MDS1(config)# show islb initiator config
Node WWN is 21:01:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:02:00:05:9b:70:5a:42
Port WWN is 21:03:00:05:9b:70:5a:42
Node WWN is 21:04:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:05:00:05:9b:70:5a:42
Port WWN is 21:06:00:05:9b:70:5a:42
Node WWN is 21:07:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:08:00:05:9b:70:5a:42
Port WWN is 21:09:00:05:9b:70:5a:42
Node WWN is 21:0a:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:0b:00:05:9b:70:5a:42
Port WWN is 21:0c:00:05:9b:70:5a:42
Node WWN is 21:0d:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:0e:00:05:9b:70:5a:42
Port WWN is 21:0f:00:05:9b:70:5a:42
MDS1(config)#
330
The next 2 questions state that we need to configure the first iSLB virtual-target,
however we are not allowed to use the command required for this. Remember that you also
can not use any zoning changes to allow the traffic of the iSCSI initiator to the FC
target.
In this question we will be using the so-called initiator targets. These are targets that
are directly configured under the iSLB initiator and automatically only this initiator
can access the target. This eases the configuration of your initiators and it adds another
option for flexibility when you are configuring some disk for just one initiator. It will
automatically arrange the zoning for you by using the auto-zoning feature.
In this case its not really efficient to use the initiator-target feature, because we have a
single target that is reached by multiple initiators.
Read ahead as you need to make this target high available with another target and use LUN
masking!
MDS2
MDS2(config-islb-init)# ?
metric
Assign load balancing metric for the initiator
no
static
target
Configure iSLB initiator target
username
Assign username for iSLB login authentication
vsan
zonename
Assign zone name for the initiator targets
end
Go to exec mode
exit
pop
push
where
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 ?
<CR>
fc-lun
Fc-lun of the fc-target
iqn-name
Name of the target
no-zone
No automatic zoning
revert-primary-port Revert back to primary port when it comes back up
sec-device-alias
Device-alias of the secondary fc-target
sec-pwwn
PWWN of the secondary fc-target
trespass
Enable trespass support
We need to configure this target in VSAN 10. Do not forget to add the initiator to VSAN
10 as well. Either by configuring the iSCSI interface inside this VSAN, which we already did, or
by configuring the VSAN command under the initiator.
When you do not do this, the initiator will not be active in this zone.
MDS2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 ?
331

iscsi-lun
ISCSI-lun of the target
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun

<CR>
iqn-name
Name of the target
no-zone
No automatic zoning
sec-device-alias
sec-pwwn
trespass
sec-device-alias J2D2 ?
MDS2(config-islb-init)#
sec-device-alias J1D2 sec-lun 0x0 ?
<CR>
iqn-name
Name of the target
no-zone
No automatic zoning
sec-vsan
Assign VSAN membership for the initiator target
trespass
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 ?
<CR>
Range separator
iqn-name
Name of the target
no-zone
No automatic zoning
0xa ?
0xa
0xa
0xa
Now we configured the required LUN masking. The FC LUN 0x0 should be seen as 0xA.
MDS2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x1 iscsi-lun 0xb
You can configure the same IQN name, when you only change the LUN masking configuration.
This way your target is visible with 2 LUNs. This will also be visible in the show outputs.
MDS2
MDS2(config-islb-init)# islb zoneset activate
MDS2(config)# islb commit
The final command required is to make the auto-zoning active. This is not strictly necessary
as with CFS distribution enabled, the auto-zoning is automatically activated as soon as
the commit command is entered.
The commands for the initiator-targets are very lengthy, which might cause issues with
your terminal emulator. When you are running into issues, you can enable a longer
terminal width.
MDS2(config)# sh run | section islb
islb distribute
static nWWN 21:01:00:05:9b:70:5a:42
static pWWN 21:02:00:05:9b:70:5a:42
332

static pWWN 21:03:00:05:9b:70:5a:42
vsan 10
target pwwn c6:83:00:11:0d:18:fb:ce vsan 10 fc-lun 0x0000 iscsi-lun 0x000a sec
-pwwn c6:83:00:11:0d:18:fb:ce sec-vsan 10 sec-lun 0x0000 iqn-name iqn.jbod-disk2
-islb
target pwwn c6:83:00:11:0d:18:fb:ce vsan 10 fc-lun 0x0001 iscsi-lun 0x000b sec
-pwwn c6:83:00:11:0d:18:fb:ce sec-vsan 10 sec-lun 0x0001 iqn-name iqn.jbod-disk2
-islb
static nWWN 21:04:00:05:9b:70:5a:42
static pWWN 21:05:00:05:9b:70:5a:42
static pWWN 21:06:00:05:9b:70:5a:42
static nWWN 21:07:00:05:9b:70:5a:42
static pWWN 21:08:00:05:9b:70:5a:42
static pWWN 21:09:00:05:9b:70:5a:42
metric 2000
static nWWN 21:0a:00:05:9b:70:5a:42
static pWWN 21:0b:00:05:9b:70:5a:42
static pWWN 21:0c:00:05:9b:70:5a:42
static nWWN 21:0d:00:05:9b:70:5a:42
static pWWN 21:0e:00:05:9b:70:5a:42
static pWWN 21:0f:00:05:9b:70:5a:42
islb commit
islb vrrp 71 load-balance
MDS2(config)#
Although the device-alias mode is enhanced. They are not kept in the configuration, but
are translated to the pWWN of the device-alias.
MDS2(config)# show islb initiator config
Node WWN is 21:01:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:02:00:05:9b:70:5a:42
Port WWN is 21:03:00:05:9b:70:5a:42
Initiator Target: iqn.jbod-disk2-islb
Port WWN c6:83:00:11:0d:18:fb:ce [J1D2]
Secondary PWWN c6:83:00:11:0d:18:fb:ce [J1D2]
Primary PWWN VSAN 10
Secondary PWWN VSAN 10
iSCSI LUN: 0x000a, FC LUN: 0x0000, Secondary FC LUN: 0x0000
iSCSI LUN: 0x000b, FC LUN: 0x0001, Secondary FC LUN: 0x0001
Zoning support is enabled
333
This verification command shows you in summary how the initiator target is automatically
configuring zones and how the 2 LUNs are visible and summarized in this single target.
Now we can configure the remaining initiators for the same target.
MDS2
MDS2(config-islb-init)# target device-alias J2D2 vsan 10 fc-lun 0x0 iscsi-lun 0xa
vsan 10 fc-lun 0x1 iscsi-Configuring iSLB initiator target with device-alias J2D2
(VSAN 10) and sec-device-alias J1D2 (VSAN 10) fails: Initiator target name invalid
or duplicate (0x403c0044)
Now although this target is only available for the initiator where it is configured. You still
can not have overlapping symbolic-names in the configuration! Therefore we will add the
host number to the symbolic-name.
MDS2
sec-device-alias J1D2 sec-lun 0x0 sec-vsan 10 iqn-name iqn.jbod-disk2-islb2
MDS2(config-islb-init)# vsan 10
MDS2(config-islb-init)# exit
0xa
0xb
0xa
0xb
0xa
0xb
0xa
0xb
Now we can verify our lengthy configuration!

MDS2(config)# show islb initiator configured
Member of vsans: 10
Node WWN is 21:01:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:02:00:05:9b:70:5a:42
Port WWN is 21:03:00:05:9b:70:5a:42
334

Initiator Target: iqn.jbod-disk2-islb
Port WWN c6:83:00:11:0d:18:fb:ce [J1D2]

Node WWN is 21:04:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:05:00:05:9b:70:5a:42
Port WWN is 21:06:00:05:9b:70:5a:42
Initiator Target: iqn.jbod-disk2-islb2
Port WWN 21:00:00:11:0d:18:ff:ce [J2D2]

Member of vsans: 10
Node WWN is 21:07:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:08:00:05:9b:70:5a:42
Port WWN is 21:09:00:05:9b:70:5a:42
Port WWN 21:00:00:11:0d:18:ff:ce [J2D2]
335

Member of vsans: 10
Node WWN is 21:0a:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:0b:00:05:9b:70:5a:42
Port WWN is 21:0c:00:05:9b:70:5a:42
Port WWN 21:00:00:11:0d:18:ff:ce [J2D2]

Member of vsans: 10
Node WWN is 21:0d:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:0e:00:05:9b:70:5a:42
Port WWN is 21:0f:00:05:9b:70:5a:42
Port WWN 21:00:00:11:0d:18:ff:ce [J2D2]
MDS2(config)#
Now we have configured our 2 LUNs to be highly available using auto-zoning in all
configured initiators!
The next 2 questions indicate another target that now should only be created under
iqn.islb-initiator-host-3. We dont need to be worried about the high availability as it
will be by default!
However we can not use auto-zoning!
MDS2
MDS2(config-vsan-db)# islb initiator name iqn.islb-initiator-host-3
336

MDS2(config-islb-init)# target device-alias J1D1 ?
<CR>
fc-lun
Fc-lun of the fc-target
iqn-name
Name of the target
no-zone
No automatic zoning
sec-device-alias
sec-pwwn
trespass
vsan
Assign VSAN membership for the initiator target
MDS2(config-islb-init)#
MDS2(config-islb-init)# target device-alias J1D1 vsan 20 no-zone iqn-name iqn.islbjbod1-disk1
Configuring iSLB initiator target with device-alias J1D1 (VSAN 20) fails: Initiator
target's primary pWWN is not in the same VSAN as the initiator (0x403c0060)
When you are configuring the initiator target, you must first configure the VSAN where
the initiator resides, before it will accept the command. You need to make the initiator
member of all VSANs that it needs to access targets in.
MDS2
MDS2(config-islb-init)# target device-alias J1D1 vsan 20 no-zone iqn-name iqn.islbjbod1-disk1
Now we are not finished as we still need to create the zoning for this new initiator
target.
Here we can not use the IP address or the Symbolic Name of the initiator in the zone.
This means we need to use the statically assigned pWWN values or the nWWN. In this case we will
use
the
pWWN
that
is
assigned
by
the
MDS
switch.
MDS2
MDS2(config)# zoneset name ZS1 v 20
Enhanced zone session has been created. Please 'commit' the changes when done.
MDS2(config-zoneset)# zone name ISLB_1
MDS2(config-zoneset-zone)# show islb initiator configured iqn.islb-initiator-host-3
Member of vsans: 10, 20
Node WWN is 21:07:00:05:9b:70:5a:42
No. of PWWN: 2
Port WWN is 21:08:00:05:9b:70:5a:42
Port WWN is 21:09:00:05:9b:70:5a:42
Port WWN 21:00:00:11:0d:18:ff:ce [J2D2]
337

MDS2(config-zoneset-zone)# member device J1D1
MDS2(config-zoneset-zone)# member pwwn 21:08:00:05:9b:70:5a:42
MDS2(config-zoneset-zone)# member pwwn 21:09:00:05:9b:70:5a:42
MDS2(config-zoneset-zone)# zoneset activate name ZS1 v 20
MDS2(config)# zone commit v 20
When you are using the pWWN for the zoning, keep in mind to add both of the pWWNs as the
initiator will get one out of 2 assigned, so you have to take care that either one can be the
active pWWN at that moment.
Now the final question of this task and this chapter is to configure authentication for all
initiators. This means that we need to configure the CHAP authentication on the
initiator.
338
MDS2
MDS2(config-islb-init)# username host-1
MDS2(config)# username host-1 password iSLBpassw0rd iscsi
To finish we create the local authentication database entries, just like was required for the
iSCSI configuration.
Here we finished the chapter about Storage Networking Extensions over IP!
The next Chapter will do a deep dive into the subject of Unified Fabric. Where we will
focus on Fibre Channel technologies over Ethernet. Which are very different than the FCIP
technologies that we used in this chapter.
Take a break and be fresh to start the next chapter!
339
Chapter 7: Data
Center Unified
Fabric
Chapter 7: Data Unified Fabric is intended to let you be familiar with the Storage Networking features
available on the Cisco Nexus switches and combined with the Cisco MDS switches.
This chapter will be about implementing FCoE features inside of the Nexus switches and the backwards
compatibility with Native FC connections. Besides that we will be looking at N-Port Virtualization
configurations..
on your own diagram, making it much easier when you step into the real lab. Multiple topology
340
General Rules
particular chapter
2 hours
341
Solutions
Task 1: Native Fibre Channel on Nexus
In this chapter we will be configuring several Unified Fabric features related to the Cisco Nexus
switches and the Cisco MDS platform. During this chapter we will be configuring features
related to transporting Fibre Channel across Ethernet connections. This technology is called
Fibre Channel over Ethernet (FCoE).
The first questions in this initial task are to create some initial configuration. We need to make
sure that we leave configurations in tact on MDS1 and MDS2, because we will be re-using the
topology that we build in previous chapters.
The topology we will be using throughout this chapter is as the following drawing illustrates.
We will not be using VDCs in this topology, but only the native switching. Except for some
special feature which we will be talking about in the next couple tasks.
The topology is using the 3 Nexus switches, the 2 MDS switches and the 2 JBOD arrays so we
can create a Fibre Channel fabric in combination with the Ethernet network of our lab.
This combination is the Cisco best practice for Datacenter networking, where the Distribution
switches (Nexus 5500) are connecting to both the Core switch (Nexus 7000) and Fibre Channel
Fabric (MDS switches).
342
We will be re-using the MDS configurations except that we need to disable the
GigabitEthernet interfaces as we do not need the FCIP connections anymore, which might
interfere with the next questions.
MDS1
MDS1(config-if)# shutdown
MDS2
Now we need to extend our Fibre Channel fabric with native interfaces towards the Nexus 5500
switches.
The Nexus 5500 is the only switch in the Nexus series that supports native Fibre Channel
interfaces, besides the UCS 61xx fabric interconnects (which are not fully Nexus switches). The
first generation of Nexus 5000 switch (Nexus 5010 and Nexus 5020) supported FC uplinks
using dedicated uplink modules with either 10Gbps Ethernet (or FCoE) ports or dedicated
8Gbps Fibre Channel interfaces. These Fibre Channel interfaces are auto-switching for 8Gbps,
4Gbps, 2 Gbps and 1Gbps FC traffic, depending on the module (some only support up to
4Gbps).
With the second generation of Nexus 5000 series. Cisco introduced the concept of universal
ports. These universal ports are by default enabled for Ethernet and FCoE traffic, but
can be converted with a single command to native Fibre Channel interfaces. Of course you do
need to change the SFP in the port, before it will actually work. Fibre Channel has different
SFPs than Ethernet SFPs.
The interfaces are not licensed, but before you are able to use the Fibre Channel ports on the
Nexus 5500 chassis, you need to have a Storage Protocols Services License. This
license is valid for enabling both native Fibre Channel interfaces and FCoE on any of the ports in
the chassis.
You need to enable the Fibre Channel ports in the switch and this is a disruptive change! Every
additional Fibre Channel that you configure requires a reboot of the whole switch. This is
because the ASIC that is controlling the ports needs to be re-initialized and reprogrammed for
Fibre Channel traffic on that port.
There are limitations to which ports can be enabled for Fibre Channel traffic and which are not.
It means that you need to configure Fibre Channel ports next to each other. Its not possible to
configure one interface as Ethernet and the next interface as Fibre Channel.
343
The following drawing illustrates an example in which some of the ports are enabled for Fibre
Channel and other ports are enabled for Ethernet, which is how you might want to cluster
certain ports that are serving a certain system.
The combination of FC and Ethernet is not possible to configure.
What is possible with the Unified Port technology is that the interfaces which can be
configured for Fibre Channel are on the right side of the switch. Meaning that all first ports can
be configured for Ethernet and the last ports for Fibre Channel.
This can not be mixed or turned around. There is no technical explanation for it, just was a
design decision of Cisco to do it this way.
The port layout on the switch will be like the following drawing illustrates.
You are able to configure from a single port to all ports as Fibre Channel ports, there is no
limitation to the amount of ports, just that they need to be next to each other.
Now we need to configure the Unified Ports on our switches.
Do not forget to do the reboot of the switch to make them active.
SW2
SW2(config)# slot 1
SW2(config-slot)# port 31-32 type fc
SW2(config-slot)# wry
[########################################] 100%
SW2(config-slot)# reload
344

WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
SW3
SW3(config)# slot 1
SW3(config-slot)# port 31-32 type fc
SW3(config-slot)# wr
[########################################] 100%
SW3(config-slot)# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
After the reboot and you issue a show interface brief, you will see 2 new fc1/31 and
fc1/32 in your configuration to be configured. The SFPs in the ports are already of the Fibre
Channel type. So they will be immediately recognized and you are able to configure the native
Fibre Channel connection to the MDS switches.
Now when we read ahead in the number of questions we see that the interfaces should be
combined into a single connection as the FSPF protocol is only allowed to see a single
connection in its database. This means that we need to configure a port-channel.
Just like its possible to create ethernet port-channels, its also possible to configure Fibre
Channel port-channels just like on the MDS switches.
We first create our configuration for the MDS switches before the Nexus switches.
The configuration on the MDS switches is just equal to any other port, as the Nexus just acts as
a standard Fibre Channel Director/Switch like all other MDS switches. NX-OS is also a
rebranding of SAN-OS, so all FC code is still included in the OS and is not different in CLI than
the MDS switches.
MDS1
MDS1(config-if)# switchport trunk allowed vsan 10
MDS1(config-if)# switchport trunk allowed vsan add 20
MDS2
345

MDS2(config-if)# switchport trunk allowed vsan 10
MDS1(config-if)# switchport trunk allowed vsan add 20
Do not forget to enable the port-channel interface in another VSAN, because we are still using
the configuration from the other chapters the Default VSAN is still suspended and not able to
enable ports in.
Then we configure the Nexus switches so the port-channels will come online.
SW2
SW2(config)# int fc1/31
SW2(config-if)# channel-group 10
command failed: port not compatible [port mode]
Interfaces that are set to their default auto-mode are not allowed to be configured in a portchannel. This means that we first need to manually set the port-mode before we can enable the
port in a port-channel
SW2(config-if)# sw mode e
please do the same operation on the switch at the other end of the port-channel,
SW2(config-if)# int fc1/32
SW2(config-if)# int port-channel10
Channel group 10 is already a SAN port channel
^
346
The port-channel configuration on the Nexus 5000 series switches is not equal to the MDS
configuration for Ethernet and FC port-channels. You must use unique numbering for all portchannels (both Ethernet and FC), but you cannot configure them with the same configuration.
For FC port-channels you need to use the special command san-port-channel instead of
the regular port-channel command.
There you have the configuration available that you would expect to have with Fibre Channel
port-channels.
SW2(config-if)# int san-port-channel 10
SW2(config-if)# sw trunk mode on
SW2(config-if)# sw trunk allowed vsan 10
SW2(config-if)# sw trunk allowed vsan add 20
SW2(config-if)# vsan database
SW2(config-vsan-db)# vsan 10
SW2(config-vsan-db)# vsan 10 interface san-port-channel 10
SW3
SW3(config)# int fc1/31
SW3(config-if)# int fc1/32
SW3(config-if)# int san-port-channel 10
SW3(config-if)# sw trunk mode on
SW3(config-if)# sw trunk allowed vsan 10
SW3(config-if)# sw trunk allowed vsan add 20
SW3(config-vsan-db)# vsan 10 interface san-port-channel 10
Now we configured both Nexus switches inside their own SAN Port-channel and gave them
membership to another VSAN, just to keep the configuration equal to the MDS switches to
prevent failures in the traffic when interfaces come online.
First we check if the interfaces are really seen in the Nexus chassis. This is an output of a preconfigured Nexus 5000 switch.
SW2(config-if)# do sh int brief
347

------------------------------------------------------------------------------Interface Vsan
Admin Admin
Status
SFP
Oper Oper
Port
Mode
Trunk
Mode Speed Channel
Mode
(Gbps)
------------------------------------------------------------------------------fc2/1
4094
E
on
sfpAbsent
--10
fc2/2
4094
E
on
sfpAbsent
--10
fc2/3
4094
F
on
sfpAbsent
---fc2/4
4094
F
on
sfpAbsent
---fc2/5
4094
F
on
sfpAbsent
---fc2/6
4094
F
on
sfpAbsent
---fc2/7
4094
F
on
sfpAbsent
---fc2/8
4094
F
on
sfpAbsent
----------------------------------------------------------------------------------Ethernet
VLAN
Type Mode
Status Reason
Speed
Port
Interface
Ch #
-------------------------------------------------------------------------------Eth1/1
1
eth trunk up
none
1000(D) 3
Eth1/2
1
eth access down
SFP not inserted
10G(D) -Eth1/3
1
eth access down
SFP not inserted
10G(D) -Eth1/4
1
eth access down
SFP not inserted
10G(D) -Eth1/5
1
eth access up
none
10G(D) -Eth1/6
1
eth access up
none
10G(D) -Eth1/7
1
eth trunk up
none
10G(D) -Eth1/8
1
eth access down
SFP not inserted
10G(D) -Eth1/9
1
eth trunk up
none
10G(D) -Eth1/10
1
eth trunk up
none
10G(D) -Eth1/11
1
eth access down
SFP not inserted
10G(D) -Eth1/12
1
eth access down
SFP not inserted
10G(D) -Eth1/13
1
eth access down
SFP not inserted
10G(D) -Eth1/14
133
eth access up
none
10G(D) -Eth1/15
10
eth access down
SFP not inserted
10G(D) -Eth1/16
1
eth access down
SFP not inserted
10G(D) -Eth1/17
1
eth trunk down
SFP not inserted
10G(D) 17
Eth1/18
1
eth trunk down
SFP not inserted
10G(D) 18
Eth1/19
1
eth trunk up
none
10G(D) 4096
Eth1/20
1
eth access up
none
10G(D) --------------------------------------------------------------------------------Interface
Vsan Admin Status
Oper Oper
IP
Trunk
Mode Speed
Address
Mode
(Gbps)
-------------------------------------------------------------------------------san-port-channel 10
4094 on
noOperMembers ---------------------------------------------------------------------------------Port-channel VLAN Type Mode
Status Reason
Speed Protocol
Interface
-------------------------------------------------------------------------------Po3
1
eth trunk up
none
a-1000(D) lacp
Po17
1
eth trunk down
No operational members
auto(D) none
Po18
1
eth trunk down
10G(D) lacp
Po202
1
eth trunk down
auto(D) lacp
Po1234
1
eth trunk down
10G(I) none
Po4096
1
eth trunk up
none
a-10G(D) lacp
-------------------------------------------------------------------------------Port
VRF
Status IP Address
Speed
MTU
-------------------------------------------------------------------------------mgmt0 -up
10.160.45.14
1000
1500
348

------------------------------------------------------------------------------Interface Vsan
Admin Admin
Status
SFP
Oper Oper
Port
Mode
Trunk
Mode Speed Channel
Mode
(Gbps)
------------------------------------------------------------------------------vfc1
1
E
on
trunking
-TE
10
-SW2(config-if)#
You can see that the port-channels for both Ethernet and FC are separated from each other and
can be configured separately. The FC interfaces, although they are connected to the same ASIC
as the Ethernet interfaces are shown different from the normal Ethernet interfaces.
Then we verify our VSAN database membership.
SW2(config-vsan-db)# show vsan member
vsan 1 interfaces:
vsan 10 interfaces:
fc1/31
fc1/32
san-port-channel 10
vsan 20 interfaces:
vsan 4079(evfp_isolated_vsan) interfaces:
vsan 4094(isolated_vsan) interfaces:
SW2(config-vsan-db)#
Here we see that both the port-channel and the individual interfaces have members in the
VSAN that we configured it in.
Now we have a successful native FC connection between the MDS and the Nexus switches!
The next question states that we should request the lowest possible Domain ID.
In the previous chapters we configured parameters for giving out FC Domain IDs that
prevent the switches to use very low numbers. Therefore we will automatically get higher
numbers assigned by configuring the lower numbers.
SW2
SW2(config)# fcdomain domain ?
<0-239>
Configure the domain id in decimal, 0 not allowed for static dom
<0x0-0xef> Configure the domain id in hexadecimal
SW2(config)# fcdomain domain 0 ?
preferred Configure the domain id as preferred
static
Configure the domain id as static
SW2(config)# fcdomain domain 0 preferred ?
vsan Specify the vsan range
SW2(config)# fcdomain domain 0 preferred vsan 10
SW2(config)# fcdomain domain 0 preferred vsan 20
349
Do not forget to restart the FC Domain when you are changing Domain IDs. Without a restart
the domain IDs will not be effective!
SW2
SW2(config)# fcdomain restart disruptive vsan 10
SW2(config)# fcdomain restart disruptive vsan 20
SW3
SW3(config)#
SW3(config)#
SW3(config)#
SW3(config)#
fcdomain
fcdomain
fcdomain
fcdomain
domain 0 preferred
domain 0 preferred
restart disruptive
restart disruptive
vsan
vsan
vsan
vsan
10
20
10
20
Now the switches will have assigned a higher number FC Domain ID, but have configured the
lowest possible value, which is 0.
Now we should be able to see all devices that are located in VSAN 10 and VSAN 20. Meaning
we are able to see all the JBOD disks in both VSANs and we should have zoning applied as this is
also taken care of when fabrics merge.
The final task is that we should keep in mind that there are already security mechanisms in
place in the VSANs. Both VSAN 10 and VSAN 20 are using port-security. So we should add
the Nexus switches SWWNs to those databases before they are fully operational!
We dont have any hosts connected to the Nexus switches so the changes to the portsecurity database is not that impacting as we only need to add the switch WWNs to the
port-security database to add the switches to the fabric.
Lets check the port-security database of VSAN 10 on both MDS1 and MDS2 that we
configured in previous chapters.
MDS1
MDS1(config)# do sh port-sec database
-------------------------------------------------------------------------------VSAN Logging-in Entity
Logging-in Point
(Interface)
-------------------------------------------------------------------------------10
10
c6:81:00:11:0d:02:01:07(pwwn) 20:04:00:0d:ec:6f:4f:40(fc1/4)*
10
10
c6:80:00:11:0d:01:01:07(pwwn) 20:03:00:0d:ec:6f:4f:40(fc1/3)*
10
20:01:54:7f:ee:05:08:79(swwn) 20:05:00:0d:ec:6f:4f:40(fc1/5)*
10
20:01:54:7f:ee:05:08:79(swwn) 20:06:00:0d:ec:6f:4f:40(fc1/6)*
10
20:00:00:a3:bf:33:11:33(pwwn) 20:0b:00:0d:ec:6f:4f:40(fc1/11)*
10
20:00:00:a3:fe:00:98:32(pwwn)
Any
[Total 8 entries]
MDS2
MDS2(config-port-security)# do sh port-sec database
-------------------------------------------------------------------------------Copyright by IPexpert. All rights reserved.
350

VSAN Logging-in Entity
Logging-in Point
(Interface)
-------------------------------------------------------------------------------10
21:01:00:e0:8b:3c:c0:b5(pwwn) 20:01:54:7f:ee:05:08:78(fc1/1)*
10
20:00:00:0d:ec:6f:4f:40(swwn) 20:05:54:7f:ee:05:08:78(fc1/5)*
10
20:00:00:0d:ec:6f:4f:40(swwn) 20:06:54:7f:ee:05:08:78(fc1/6)*
10
20:00:00:a3:de:11:66:2b(pwwn) 20:00:54:7f:ee:05:08:78(swwn)
10
20:00:00:a3:fe:00:98:32(pwwn)
Any
[Total 5 entries]
Here we see that VSAN 10 is not using a distributed port-security database, so we need
to add the Nexus switches to both of the databases on the switches.
We will do this from the MDS point of view, again because its not distributed.
Check the SWWN of the 2 switches:
SW2
SW2# show wwn sw
Switch WWN is 20:00:00:05:9b:73:1b:c0
SW3
SW3# show wwn sw
Switch WWN is 20:00:00:05:9b:a4:1f:aa
Now we add those entries to the connecting MDS switches. In our topology SW2 is connected
to MDS1 and SW3 is connected to MDS2.
MDS1
MDS1(config-port-security)# swwn 20:00:00:05:9b:73:1b:c0
MDS2
MDS2(config-port-security)# swwn 20:00:00:05:9b:a4:1f:aa
Now the SWWN of the switches has been added to the port-security database and you are
good to go and have a working native Fibre Channel fabric for VSAN 10.
Now we shouldnt forget about VSAN 20 as well, as this VSAN is also allowed on the trunk to
the MDS switches.
The port-security of VSAN 20 is distributed, so the change only needs to be done from one
MDS switch. In the previous chapter we were only allowed to do configuration changes on
MDS1 so we will honor this again in this question.
MDS1
-------------------------------------------------------------------------------VSAN Logging-in Entity
Logging-in Point
(Interface)
-------------------------------------------------------------------------------Copyright by IPexpert. All rights reserved.
351

20
c6:80:00:11:0d:07:fa:ce(pwwn)
20
c6:80:00:11:0d:01:01:07(pwwn)
20
c6:81:00:11:0d:17:fa:ce(pwwn)
20
c6:81:00:11:0d:02:01:07(pwwn)
20
21:01:00:e0:8b:3c:c0:b5(pwwn)
[Total 5 entries]
MDS1(config)#
20:03:00:0d:ec:6f:4f:40(fc1/3)*
20:03:00:0d:ec:6f:4f:40(fc1/3)*
20:04:00:0d:ec:6f:4f:40(fc1/4)*
20:04:00:0d:ec:6f:4f:40(fc1/4)*
20:01:54:7f:ee:05:08:78*
Now on to the configuration of our database.

MDS1
MDS1(config)# port-security
port-security database vsan 20

swwn 20:00:00:05:9b:73:1b:c0
swwn 20:00:00:05:9b:a4:1f:aa
port-security activate vsan 20
commit v 20
After activating the database on the MDS1, we can see an activated database on both MDS
switches with all necessary information for activating the security in VSAN 20.
Now we have finished our first task in this chapter by configuring native FC interfaces on the
Nexus series switches.
352
Task 2: Fibre Channel over Ethernet (FCoE)
In this second task we will be configuring a unique feature of the Cisco Nexus platform. The
switches have the possibility to transport Fibre Channel frames across Ethernet links.
This is done so you only require one cable instead of two to your server that needs to connect
to devices on the Fibre Channel Fabric. This technology is also called Unified Fabric.
In your server you require a Converged Network Adapter (CNA). This CNA has a chip to
process both Fibre Channel and Ethernet frames. The processing occurs simultaneously.
The initial version of FCoE required a fixed configuration of a port to do FCoE. This has changed
in a newer version of the protocol, where a helper protocol was introduced called Datacenter
Bridging Exchange Protocol (DCBX). This protocol negotiates the features supported by
the switch and the connecting server with a CNA.
The switch is always leading in its communication. When the switch communicates a certain
configuration value, like the VLAN number. This prevents miscommunications on the server
side. So the networking team can direct the server what values to use.
The protocol is an extension to the widely used LLDP protocol and is also called the FCoE
Initialization Protocol (FIP).
There are 2 very different types of FCoE. The first is the standard F-port virtualization.
Where you connect your Fibre Channel environment to the Converged Network Adapter
(CNA) of a server. The Nexus then acts as a distribution switch for the first hop which is
converged and the next hop, the traffic is split in both an Ethernet uplink and a Fibre Channel
uplink.
The second type is the E-port virtualization. This is also called multi-hop FCoE as you are
no longer converging only the first hop, but also further hops in the network. You can connect
Nexus switches using virtual E-ports to extend the Fibre Channel fabric through your Ethernet
network. In fact, you do not need a MDS switch in some topologies as they can perform all
features necessary, of course depending on the port density of your native Fibre Channel
environment.
In this first task about FCoE we will be using the converged F-Ports.
The initial question is related to the fact that we need to configure a server which is connected
to interface Ethernet1/24. We want to use a port-channel from the server to the network
and we are using the vPC technology for this use.
First we need to ensure that the vPC technology works again across the Nexus switches. We
will use the easiest solution, but configuring peer-keepalives across the mgmt0 interface
and one of the interlinks of the Nexus switches as the peer-link.
We did not get any requirements in this question so we can use any values we want!
353
You might see more of these types of questions in the lab, where you are free to use values you
want for the required configuration for this question. It lets you think about all the changes that
are necessary to complete the question.
We will configure the switches for the vPC feature first.
SW2
SW2(config-vpc-domain)# peer-keepalive destination 172.16.100.3
As we are using the mgmt0 interfaces as a source for the keepalives we do not need to
configure the source of the keepalives as its using the mgmt0 and the management VRF by
default as a source for the keepalives.
SW3
SW3(config-vpc-domain)# peer-keepalive destination 172.16.100.2
Now we need to finish the vPC peer configuration by configuring the peer-link. Keep in mind
that the peer-link needs to be a port-channel. In this case we only add one interface to it, as
we only require some communication.
SW2
SW2(config)# interface ethernet1/7
SW3
SW3(config)# interface ethernet1/7
Lets verify if the switches see each other and are able to form vPCs.
SW3(config-if)# sh vpc
Legend:
vPC domain id
:
Peer status
:
:
Configuration consistency status:
:
:
vPC role
:
:
Peer Gateway
:
101
peer is alive
success
success
success
secondary
0
Disabled
354

: : Enabled

--------------------------------------------------------------------id
Port
Status Active vlans
---------- -------------------------------------------------1
Po1001 up
1,100
Now we finish the first questions by configuring the vPC connecting the host.
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int e1/24
sw mode trunk
no shut
channel-group 1002
int po1002
sw mode trunk
no shut
vpc 1002
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int e1/24
sw mode trunk
no shut
channel-group 1002
int po1002
sw mode trunk
no shut
vpc 1002
Our vPC is online and is able to process traffic just like any other vPC!
Now we are going to configure the FCoE specifics. One very important topic to always keep in
mind is that you are required to have a strict distinction between Fibre Channel fabrics!
The Fibre Channel world is arranged 2 fabrics that are in no place connected to each other. This
has to do with the auto-learning nature of the Fibre Channel Network that devices can
interfere with each other when something goes wrong.
The hosts in the Fibre Channel network are smart enough to detect failures and failover to the
other fabric when this is necessary. This means that even we do have a virtual Port-Channel to
the connecting server, we still are not going to share Fibre Channel configuration across this
link.
The FC traffic is always carried in a dedicated VLAN. You cannot carry multiple VSANs over a
single VLAN. There is always a distinct VSAN to VLAN mapping. This mapping of course
needs to be equal on all switches in the topology. In this case we will not be using the 2 VSANs
across the switches, but we will keep the configuration of the VSAN to the local switch.
355
SW2
SW2(config)# feature fcoe
SW2(config-vlan)# vlan 10
SW2(config-vlan)# fcoe vsan 10
SW3
SW3(config)# feature fcoe
Next we configure the vPC interface for the FCoE traffic. We do this by first creating a virtual
interface. This virtual Fibre Channel interface (VFC) will be handling all Fibre
Channel communication.
To ensure you are only using one leg of the vPC you need to bind the individual interface
instead of the port-channel interface to this virtual FC interface.
SW2
SW2(config-vlan)# interface vfc1001
SW2(config-if)# bind interface Ethernet1/24
SW2(config-vsan-db)# vsan 10 interface vfc1001
SW3
SW3(config-vlan)# interface vfc1001
SW3(config-if)# bind interface Ethernet1/24
This configuration enables the connection between the physical Ethernet interfaces. The Virtual
FC interface and the VSAN in which it belongs. Now the switch knows that it needs to use which
VLAN to tag the traffic when sending it across the physical interface.
We have a clear distinction across the Fibre Channel fabric, but still the Ethernet traffic is using
a virtual Port-Channel and can still load balance across the 2 interfaces.
Next we need to ensure that the switches discard frames that are not from their own fabric as
we are creating different FC fabrics.
This is done using the FC-MAP feature of the Nexus. By default the Nexus switches use a value
of 0E.FC.00 as the Fabric Identifier. When a switch does not belong to a fabric (so the
FC-MAP value is different) the switch discards frames.
356
You are able to configure FC-MAP values in the range of 0E.FC.00 up to 0E.FC.FF.
We will configure a different FC-MAP value on SW2 to ensure that it will drop traffic coming
from SW3.
SW2
SW2(config)# fcoe fcmap ?
<0xefc00-0xefcff> Enter FCMAP
SW2(config)# fcoe fcmap 0x0efc02
Warning: Changing the FC-MAP will flap all VFC interfaces. Continue [y/n]? [no] y
SW2(config)#
The CLI help can be misleading as the 0 in front of the FC-MAP value needs to be typed in, but
the context help might indicate that its not necessary.
Next we are told to configure SW2 to be the primary switch. While the Fibre Channel fabrics can
be used independently. Its still possible to configure the CNA of the Server to connect to one
particular fabric over the other switches.
This is controlled using the FCF priority. FCF stands for Fibre Channel Forwarder,
meaning a Fibre Channel Switch. This FCF priority is by default set to 128 and the lower
the value the more preferred the switch becomes.
Therefore we will lower this priority on SW2 to a very low value to ensure that its chosen to
become the preferred FCF switch in the fabric.
We can verify the settings by using a single show command.
SW2(config)# show fcoe
Global FCF details
FCF-MAC is 00:05:9b:73:1b:c0
FC-MAP is 0e:fc:02
FCF Priority is 1
FKA Advertisement period for FCF is 8 seconds
SW2(config)#
Here we see that we configured the FC-MAP and the FCF Priority like it it supposed to be.
The next task would be easily solvable by only allowing the FCoE VLANs on the trunk, but we
are not allowed to do this. The final question in this task is related to a feature specific to the
DCBX protocol.
The DCBX protocol can send messages regarding the VLANs that do not have a FCoE VSAN
enabled on them.
You do this by basically shutting down the LAN part of the interface.
SW2
SW2(config-if)# shutdown lan
357
SW3
SW3(config-if)# shutdown lan
By configuring this single command you signal to the CNA that only a few VLANs are allowed on
this trunk, which are enabled for FCoE. The other VLANs are not allowed to transfer any
packets across the link and the link basically became a Fibre Channel interface, with Ethernet
encapsulation!
Next we are going to use the FCoE feature for the multi-hop purpose.
Task 3: Multi hop FCoE
The third task will be regarding the configuration of the FCoE feature to continue the FCoE
path after the initial hop which is connected to the CNA of the server.
We will be disabling all ISL links on the MDS switches to ensure only the Nexus switches will be
used for ISL traffic. We want to use the Nexus switches with multiple hops in between to form
one large fabric where all Fibre Channel traffic is sent across.
This is the initial question and relatively easy to do.
We configured multiple port-channels, but we can shutdown the FC links that connect the 2
MDS switches.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
interface
shutdown
interface
shutdown
interface
shutdown
interface
shutdown
fc1/1
interface
shutdown
interface
shutdown
interface
shutdown
interface
shutdown
fc1/1
fc1/2
fc1/3
fc1/4
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
fc1/2
fc1/3
fc1/4
358
Now we broke our Fibre Channel fabric. This task is all about fixing this topology. In this task we
will be extending VSAN 20 across the Nexus switches.
In this case we will be using the same FCoE technology as we used in the previous task except
that in this case we will not be configuring F-ports, but E-ports to extend the fabric across
the Nexus switches.
There is no real other stuff happening other than the previous task where we configured a vPC
as an F-Port.
We need to ensure that VSAN 20 keeps functioning in the way that is shown in Drawing 2.
This means that we cannot use the interlink between SW2 and SW3 for traffic and we need to
use SW1 as well.
The native FC connections connecting SW2, SW3 and the MDS switches already works and VSAN
20 is working across this connection.
This means that we already have a good connection with our current Fibre Channel Fabric.
MDS1(config-port-security)# do sh flogi database vsan 20
--------------------------------------------------------------------------INTERFACE VSAN
FCID
PORT NAME
NODE NAME
--------------------------------------------------------------------------fc1/3
20
0x0a00e8 c6:80:00:11:0d:07:fa:ce c5:9a:00:06:2b:01:00:07
fc1/3
20
0x0a00ef c6:80:00:11:0d:01:01:07 c5:9a:00:06:2b:01:00:07
fc1/4
20
0x0a01e8 c6:81:00:11:0d:17:fa:ce c5:9a:00:06:2b:02:00:07
359

fc1/4
20
0x0a01ef
c6:81:00:11:0d:02:01:07
c5:9a:00:06:2b:02:00:07
Next we are configuring the FCoE virtual E-ports that will connect the Nexus switches to
each other.
360
On the Nexus 7000 the architecture of FCoE is a little different than on the Nexus 5000 series.
First of all the Nexus 7000 does not support any native FC interfaces, but only supports to be a
FCoE forwarder. The Nexus 7000 also doesnt support to be a Fibre Channel Bridge.
Which means that its not capable of encapsulating FC frames into Ethernet packets to become
FCoE frames. This of course has to do that there are no native FC packets coming into the
Nexus 7000 switch.
One other important change on the Nexus 7000 platform is that FCoE features are not
configured in the normal configuration. The control-plane of the Fibre Channel traffic is
separated from the Ethernet traffic. This is done because of the possibility to create Virtual
Device Contexts (VDCs) on the Nexus 7000 switches.
Normally you require a license to use VDCs, but when you have the FCoE license you can
enable the Storage VDC as well.
The Storage VDC or FCoE VDC works different from other VDCs. Normal VDCs require you
to allocate interfaces to it. The Storage VDC works based on the EtherType of packets.
FCoE packets have a different EtherType on which the interface knows where to direct traffic
to.
The drawing below illustrates how the Storage VDC works and separates traffic coming in over
a single link.
361
You can only create a single Storage VDC where all FCoE traffic is terminated. The use of
normal Ethernet VDCs is still supported. In this lab we will be using the global configuration
and one Storage VDC as this is the minimal configuration that is required to support FCoE on
the Nexus 7000.
To start our FCoE E-port configuration we start on the Nexus 5000s switches as they are
pretty much equal to the previous task where we configured F-port FCoE.
Drawing 2 illustrates that there are 2 connections between the Nexus 7000 and the Nexus
5000s. As the task did not explicitly state this, we shouldnt have to do this, as we could also
create 2 E-ports between the Nexus switches, but this configuration is more commonly seen.
SW2
SW2(config)# interface ethernet1/11-12
SW2(config-if)# channel-group 1002 mode active
SW2(config-if)# switchport trunk allowed vlan 20
SW2(config)# interface vfc1
SW2(config-if)# bind interface port-channel1002
SW2(config-if)# switchport mode e
SW2(config-if)# switchport trunk allowed vsan 20
SW3
SW3(config)# interface ethernet1/11-12
SW3(config-if)# channel-group 1002 mode active
SW3(config-if)# bind interface port-channel1002
SW3(config-if)# switchport mode e
SW3(config-if)# switchport trunk allowed vsan 20
After this configuration the port-channel will come only, but the VFC interfaces will remain nonoperational as the other side has not been configured yet.
Now we move on by configuring the Nexus 7000 switch. The configuration is a little different
than on the Nexus 5000.
362
SW1-1
SW1-1(config)# install feature-set fcoe
We first need to enable a complete feature-set. This set enables a number of features, one
of which is VDCs and all Fibre Channel related processes.
SW1-1
SW1-1(config)# vdc StorageVDC type storage
SW1-1(config-vdc)# system default switchport
SW1-1(config-vdc)# feature lldp
Next we configure the VDC that we want to use for Storage traffic. We need to enable LLDP in
this VDC, because this protocol is used in the FIP (DCBX) protocol to identify which features are
possible with the end host.
SW1-1
SW1-1(config-vdc)# allocate fcoe-vlan-range 20
Then we enable which VLANs that are enabled for FCoE that need to be carried through the
VDC. So instead of issuing the allocate interface command in this place, you are using the
allocate VLAN command to ensure these VLANs are connected to the VDC and FCoE traffic is
sent to this special VDC.
SW1-1
SW1-1(config-vdc)# allocate shared interface ethernet2/1-4
SW1-1(config-vdc)# allocate shared interface port-channel1-2
SW1-1(config-vdc)# exit
The final step of the VDC configuration is to add the interfaces that are used for sharing the
Ethernet and FC traffic. Additionally with the VLAN configuration you need to configure the
shared interfaces.
Your configuration options are getting limited when you issue this command and you can only
configure a plain trunk connection then on the Ethernet port and no other features.
SW1-1
SW1-1(config)# license fcoe module 2
363
The next step is to enable one of the available licenses in the chassis on the module. Without an
FCoE license enabled on the module the feature will not work. Keep in mind that the FCoE
feature only works on F-type linecards! In the CCIE Data Center lab we only use the first
generation of linecards so we have a M1 and a F1 linecard available in our chassis. In this lab we
are using the F-type line card. This is also the reason why we are using different interfaces
than we are used to on the Nexus 5000s as uplinks to the F1 linecard.
Then we configure our Ethernet interfaces with a trunk connection and the port-channel
configuration required to support the connectivity to the Nexus 5000s.
SW1-1
SW1-1(config)# interface ethernet2/1-2
SW1-1(config-if)# channel-group 1 mode active
SW1-1(config-if)# interface port-channel1
SW1-1(config-if)# switchport mode trunk
SW1-1(config-if)# switchport trunk allowed vlan 20
SW1-1(config)# interface ethernet2/3-4
SW1-1(config-if)# channel-group 2 mode active
SW1-1(config-if)# interface port-channel2
SW1-1(config-if)# switchport mode trunk
SW1-1(config-if)# switchport trunk allowed vlan 20
Now we step into the VDC and create the configurations there related to the FCoE protocol.
SW1-1
SW1-1(config)# switchto vdc StorageVDC
SW1-1-StorageVDC# conf t
Within the Storage VDC we must enter the configuration prompt again.
SW1-1
SW1-1-StorageVDC(config)# vsan database
SW1-1-StorageVDC(config-vsan-db)# vsan 20
SW1-1-StorageVDC(config-vsan-db)# exit
We create the VSAN just like we would on a normal MDS switch.

SW1-1
SW1-1-StorageVDC(config)# interface ethernet2/1-4
SW1-1-StorageVDC(config-if)# no shutdown
SW1-1-StorageVDC(config-if)# interface port-channel1-2
SW1-1-StorageVDC(config-if)# vlan 20
SW1-1-StorageVDC(config-vlan)# fcoe vsan 20
SW1-1-StorageVDC(config-vlan)# exit
364
We open up all the interfaces that are shared with this VDC and then connect the VSAN 20 to
the VLAN that is also shared with this VDC. Just like we would on the Nexus 5000s.
SW1-1
SW1-1-StorageVDC(config)# interface vfc1
SW1-1-StorageVDC(config-if)# bind interface port-channel1
SW1-1-StorageVDC(config-if)# switchport mode e
SW1-1-StorageVDC(config-if)# switchport trunk allowed vsan 20
SW1-1-StorageVDC(config-if)# vsan database
SW1-1-StorageVDC(config-vsan-db)# vsan 20 interface vfc1
SW1-1-StorageVDC(config-vsan-db)# exit
We create the Virtual E-port on a VFC interface bound to the port-channel interface
connecting to SW2.
After this configuration and we apply the VSAN database configuration we have enabled our
fabric to being extended in to SW1-1 just like we want.
SW1-1
SW1-1-StorageVDC(config-if)# bind interface port-channel2
SW1-1-StorageVDC(config-if)# switchport mode e
SW1-1-StorageVDC(config-if)# switchport trunk allowed vsan 20
SW1-1-StorageVDC(config-if)# vsan database
SW1-1-StorageVDC(config-vsan-db)# vsan 20 interface vfc2
When finishing the second interface connecting to SW3 we finished the lengty configuration.
The FCoE configuration is successfully applied and we are able to see the fabric being extended
across multiple Nexus switches.
On MDS1 we should see the JBOD connected to MDS2 and vice versa!
The next question states that we should protect against loopbacks. What is possible with VFC
connections is that interfaces are created that are looped back to the same switch.
To prevent this, you can enable a mechanism that checks for the VF ID in the FCoE packets
and when this matches to the switch its own VFID it will reject the traffic.
SW1-1
SW1-1-StorageVDC(config)# fcoe veloopback
365
The next couple questions are all related to each other. We need to configure link
authentication, just like we did in the Fibre Channel Networking chapter. This
technology is called FC-SP, where a hashed authentication key is used to send a challenge to
the other switch to authenticate the interface.
This technology is also available on the Nexus switches when you are using FCoE interfaces.
An important topic is the combination of interface modes. You can configure an interface with
different modes as it might be required that you configure an interface with optional
authentication. The table below demonstrates the different modes and what will happen when
these 2 interfaces are connected to each other.
On
Auto-Active
Auto-Passive
Off
On
Yes
Yes
Yes
Link down
Auto-Active
Yes
Yes
Yes
No authentication
Auto-Passive Yes
Yes
Off
No authentication
Link down
The On and Off modes are simple modes that do what they tell. They enforce authentication or
reject it. The auto modes are there for optional authentication. There should be at least 1
active link when both sides are on auto as the passive link will wait for an authentication
request before starting any negotiation.
Our question states that SW1-1 should never start the negotiations for authentication. The
other questions state various comments about which passwords are used.
Read these carefully as they all contain vital information to finish the task.
First look up the SWWNs of all the 3 switches so you have the information to create the
authentication entries.
SW1-1
SW1-1-StorageVDC# show wwn switch
Switch WWN is 20:00:00:05:ab:a3:16:76
SW2
SW2# show wwn sw
Switch WWN is 20:00:00:05:9b:73:1b:c0
SW3
SW3# show wwn sw
366

Switch WWN is 20:00:00:05:9b:a4:1f:aa
Now we start by configuring the passwords used to authenticate the switches towards each
other.
SW1-1
SW1-1-StorageVDC(config)# feature fcsp
SW1-1-StorageVDC(config)# fcsp dhchap ?
devicename Configure password of the remote devices
dhgroup
Configure DHCHAP Diffie-Hellman Group List (In preference)
hash
Configure DHCHAP Hash Algorithm List (In order of preference)
password
Configure local DHCHAP password
SW1-1-StorageVDC(config)# fcsp dhchap password Nexus7000password ?
<CR>
<hh:hh:hh:hh:hh:hh:hh:hh> WWN of the device with which to use this local
password
SW1-1-StorageVDC(config)# fcsp dhchap password Nexus7000password
SW2
SW2(config)# feature fcsp
SW2(config)# fcsp dhchap password SecureNexus5000-1
SW3
SW3(config)# feature fcsp
SW3(config)# fcsp dhchap password IPexpertIsAwesome
Then we continue by configuring the switches to authenticate the devices using their specified
passwords.
SW1-1
SW1-1-StorageVDC(config)# fcsp dhchap device 20:00:00:05:9b:73:1b:c0 password
SecureNexus5000-1
We authenticate SW2 with the password as specified in the question.

SW1-1
SW1-1-StorageVDC(config)# fcsp dhchap device 20:00:00:05:9b:a4:1f:aa password
IPexpertIsAwesome
367
Same accounts for SW3 which is authenticated with the specified password.
To finish we configure the interfaces to support the FC-SP configuration without actively
starting to negotiate it.
SW1-1
SW1-1-StorageVDC(config-if)# fcsp auto-passive
SW1-1-StorageVDC(config-if)# fcsp auto-passive
Next we configure SW2 and SW3 for the equal configuration.

Configuration of both switches is even equal as its using the same password for both switches.
SW2
SW2(config)# fcsp dhchap device 20:00:00:05:ab:a3:16:76 password Nexus7000password
SW2(config-if)# fcsp auto-active
SW3
SW3(config)# fcsp dhchap device 20:00:00:05:ab:a3:16:76 password Nexus7000password
SW3(config-if)# fcsp auto-active
Now our interfaces are successfully authenticated using a secure hash, but wait we should
enable a SHA-1 hash! By default the MD5 hash is enabled, so we dont comply with the full
questioning.
SW1-1
SW1-1-StorageVDC(config)# fcsp dhchap hash sha1
SW2
SW2(config)# fcsp dhchap hash sha1
SW3
SW3(config)# fcsp dhchap hash sha1
368
Now when all links are flapped again, they will be authenticated using a SHA-1 hash of the
correct key and we answered the questions correctly.
To finish this task we need to enable a mechanism so only the active switches are allowed in
the Fibre Channel Fabric of VSAN 20.
This means that we are going to use Fabric Binding. This was introduced in one of the
latest versions of NX-OS, but is now fully supported. Pay attention that Fabric Binding is only
supported in the SAN Enterprise license on the Nexus series and the Mainframe
license on the MDS switches!
Configuration is relatively easy as we configured it before in the Fibre Channel chapter.
SW1-1
SW1-1-StorageVDC# show wwn switch
Switch WWN is 20:00:00:05:ab:a3:16:76
SW1-1-StorageVDC(config)# feature fabric-binding
SW1-1-StorageVDC(config)# fabric-binding data vsan 20
SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
First add the SWWNs of the 2 MDS switches.

SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
SW1-1-StorageVDC(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
Then we add the SWWNs of the 3 Nexus switches to the database.

SW1-1-StorageVDC(config-fabric-binding)# exit
SW1-1-StorageVDC(config)# fabric-binding activate v 20
Keep in mind that you need to configure this on all the switches as the fabric-binding feature
can not be distributed using the Cisco Fabric Services (CFS), which are also available
on the Nexus switches.
SW2
SW2(config)# feature fabric-binding
SW2(config)# fabric-binding data vsan 20
SW2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
SW2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
SW2(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
SW2(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
SW2(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
SW2(config-fabric-binding)# exit
SW2(config)# fabric-binding activate v 20
SW3
SW3(config)# feature fabric-binding
SW3(config)# fabric-binding data vsan 20
SW3(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
SW3(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
369

SW3(config-fabric-binding)#
SW3(config)# fabric-binding
swwn 20:00:00:05:ab:a3:16:76
swwn 20:00:00:05:9b:73:1b:c0
swwn 20:00:00:05:9b:a4:1f:aa
exit
activate v 20
Dont forget to include the MDS switches as well, as they are still part of the fabric too!
MDS1
MDS1(config)# feature fabric-binding
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
MDS1(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
MDS1(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
MDS1(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
MDS2
MDS2(config)# feature fabric-binding
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:70:5a:00
MDS2(config-fabric-binding)# swwn 20:00:54:7f:ee:09:3e:c8
MDS2(config-fabric-binding)# swwn 20:00:00:05:ab:a3:16:76
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:73:1b:c0
MDS2(config-fabric-binding)# swwn 20:00:00:05:9b:a4:1f:aa
370
Now we completed our task and have a successful multi-hop FCoE network running for
VSAN 20. The next task will be about optimizing this network for FCoE which has some strict
QoS requirements for the network!
Task 4: FCoE Quality of Service (QoS)
This task is about optimizing the network to run FCoE services. The FCoE protocol uses a few
very strict rules which is causing the Ethernet even being called Data Center Bridging
(DCB) or Data Center Ethernet (DCE). Basically it comes down to a few changed topics
and most important within the QoS configuration.
This is related to the nature of the Fibre Channel protocol. This protocol was developed with a
lossless network in mind. The B2B credit structure caused switches to ensure that packets will
arrive at their destination without being dropped.
Now Ethernet is a very dirty medium and there is not a single guarantee when you send a
packet on it, that it will arrive on the other end. Therefore there was a solution necessary to
support Fibre Channel lossless architectures on top of the Ethernet links.
The solution for this is called the Pause frame and the No-Drop queue.
The protocol which is used in this case is the IEEE 802.3x Link Level Flow Control.
This allows hosts or switches to signal each other to pause the traffic so the host or switch can
empty its almost full buffer. This prevents tail drops (or WRED drops) as the buffer is never
exceeded in this case. This is essential for the behavior of Fibre Channel traffic when traveling
across a Ethernet link.
This technology is also called Priority Flow Control (PFC). It works together with
802.1p bits in a tagged (802.1q) frame. The protocol already communicated which CoS value
is to be used for FCoE traffic, which is also the queue in which the pause frame is active and the
no-drop behavior is honored.
By default this value is CoS 3. This can be changed, but this is not recommended.
Priority Flow Control can be enabled on a per link basis, but this is not necessary in most
scenarios as the DCBX protocol already negotiates this behavior with the connected host or
switch. This negotiation is accepted by the other end and the CoS value on which to apply PFC
is configured.
371
The first question we see is that we should make our topology to support FCoE on all Nexus
switches in which we should follow the best practices.
There are a number of options possible to configure on the Nexus 7000 switches. By default
when you enable the FCoE feature, it will not change anything about the QoS behavior of the
device. This needs to be manually changed. The Nexus 7000 has a couple of built-in QoS
behaviors that can be altered.
By default a template is active which is not honoring the No Drop behavior of CoS 3. Then
there are multiple templates available where you are able to configure an even deeper level of
Priority No Drop QoS, which might be necessary to place certain critical FCoE systems in,
so they are getting prioritized over other FCoE traffic. Or it can be used for Control traffic that
may not be dropped.
The table below demonstrates the CoS to Queue mapping on the Nexus 700 with the available
default templates.
Template
Default QoS Priority QoS
No Drop Qos
Priority No Drop QoS
default-nq-8e-policy
0,1,2,3,4
5,6,7
0,1,2,4
5,6,7
0,1,2
5,6,7
5,6,7
1,2,3
The first change we need to do is to configure the Nexus 7000 switch to actually support FCoE
queues in its configuration. By default the QoS policy on the Nexus 7000 does not incorporate a
FCoE queue. So we should enable this.
We will pick the default-nq-7e-policy for this use, as this is the most standard No Drop
configuration and we do not require any additional queues to be mapped as well.
SW1-1
SW1-1(config)# system qos
SW1-1(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
SW1-1(config-sys-qos)# exit
372
The next step is to enable the No Drop queue in the QoS configuration. The policy that is used
is one which is by default available on the Nexus 7000s. You need to enable a No Drop queue
for CoS 3 on the entire switch when enabling FCoE. Without this configuration the FCoE
feature will not work correctly and especially in terms of congestion, the FCoE traffic will not be
lossless!
One of the nice things of the Nexus 5000 series is that it automatically is enabled to support
FCoE. There are 2 QoS groups by default enabled for queueing and one is selected to have the
no-drop queue enabled.
SW2(config-if)# show queuing interface
Ethernet1/1 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0
WRR
50
1
WRR
50
RX Queuing
qos-group 0
q-size: 243200, HW MTU: 1600 (1500 configured)
drop-type: drop, xon: 0, xoff: 1520
Statistics:
Pkts received over the port
: 882892828
Ucast pkts sent to the cross-bar
: 859238738
Mcast pkts sent to the cross-bar
: 23654090
Ucast pkts received from the cross-bar : 213450997
Pkts sent to the port
: 229752814
Pkts discarded on ingress
: 0
Per-priority-pause status
: Rx (Inactive), Tx (Inactive)
qos-group 1
drop-type: no-drop, xon: 128, xoff: 240
Statistics:
:
:
:
Ucast pkts received from the cross-bar :
:
:
:
Total Multicast crossbar statistics:
Mcast pkts received from the cross-bar
0
0
0
0
0
0
Rx (Inactive), Tx (Inactive)
: 16301832
The queuing configuration is supported by the class-map and policy configuration.

The default class-maps that are available are as the following output shows.
There its shown that the FCoE traffic is by default matched on CoS 3, which can be changed.
SW2(config-if)# show class-map
Type qos class-maps

===================
class-map type qos match-any class-fcoe
373

match cos 3
class-map type qos match-any class-default
match any
class-map type qos match-any class-all-flood
match all flood
class-map type qos match-any class-ip-multicast
match ip multicast
Type queuing class-maps

=======================
class-map type queuing class-fcoe
match qos-group 1
class-map type queuing class-default
match qos-group 0
class-map type queuing class-all-flood
match qos-group 2
class-map type queuing class-ip-multicast
match qos-group 2
Type network-qos class-maps

==============================
class-map type network-qos class-fcoe
match qos-group 1
class-map type network-qos class-default
match qos-group 0
class-map type network-qos class-all-flood
match qos-group 2
class-map type network-qos class-ip-multicast
match qos-group 2
The default policy-maps are configured to match on the class-maps and to apply appropriate
queueing and MTU sizes.
SW2(config-if)# show policy-map
Type qos policy-maps

====================
policy-map type qos default-in-policy
class type qos class-fcoe
set qos-group 1
class type qos class-default
set qos-group 0
Type queuing policy-maps
========================
374
policy-map type queuing default-in-policy

class type queuing class-fcoe
bandwidth percent 50
class type queuing class-default
policy-map type queuing default-out-policy
class type queuing class-fcoe
class type queuing class-default
Type network-qos policy-maps

===============================
policy-map type network-qos default-nq-policy
class type network-qos class-fcoe
pause no-drop
mtu 2158
class type network-qos class-default
mtu 1500
Interfaces have these policies applied by default. Which makes it very easy for a QoS best
practice configuration on the Nexus 5000.
SW2(config-if)# show policy-map interface
Global statistics status :
enabled
port-channel3
Service-policy (qos) input:
policy statistics status:
default-in-policy
disabled
Class-map (qos):
Match: cos 3
set qos-group 1
class-fcoe (match-any)
Class-map (qos):
Match: any
set qos-group 0
class-default (match-any)
Service-policy (queuing) input:

default-in-policy
disabled
Class-map (queuing):
Match: qos-group 1
Match: qos-group 0
Service-policy (queuing) output:
default-out-policy
disabled
Match: qos-group 1
375
Match: qos-group 0
Now we know that QoS is enabled on the switches and that they all take care of FCoE traffic
with the No-drop queue we even included the last question of this task as well, because with
the adaption of the default QoS policies we also included the MTU statements.
Within the Nexus switches its not necessary to change the MTU size on the interfaces directly.
You can change MTU sizes per queue on the Nexus. This means that the FCoE queue can have a
different MTU than all the others.
This is automatically arranged by the templates that we apply as that the FCoE class will get the
MTU size to support maximum frame size of the Fibre Channel packet, which is 2112.
The next question about blocking receivers. The Nexus knows a special mechanism to use for
QoS and queuing. It uses a technology called Virtual Output Queuing. This technology
uses the ingress buffers of all interfaces to create a very large pool of Egress buffers.
It means that traffic is held at the ingress port, until the egress port has time to transmit the
packet. This is arranged by the token principle that the Nexus switches use. Before traffic is
sent across the switch fabric between ASICs and/or linecards. The traffic needs to have a
token before its allowed to go over the fabric. Now the packet is stalled at the ingress port,
until it receives a token to be transmitted over to the egress interface.
This Virtual Output Queueing Unicast limit is required to enable the feature that we are
looking for in this question. Which is enabled with a simple single command.
SW2
SW2(config)# hardware unicast voq-limit
376
The last question of this task introduces another important topic about FCoE traffic. The native
Fibre Channel protocol knows the concept of B2B credits (Buffer to Buffer credits)
to ensure a lossless behavior of the link. When you are using a long link on a Fibre Channel
switch, it means you require more B2B credits than before, because there are more packets
in transit on an interface at any given time, so it takes more time before acknowledgements are
received for these packets and before you are able to send additional packets onto the link.
Within FCoE there is not a concept of B2B credits, but there is the concept of Pause frames
to hold up the sending of packets until further notice.
Now this Pause frame is sent by the receiver towards the sender. When you have a long link, it
means that at the time the receiver has sent this message, the sender already sent more
packets onto the link, because there are more packets in transit. This might cause a buffer
overflow and might cause the switch to have to drop packets.
To overcome this, we need to ensure the buffer size is set to the correct size according to the
link size. The Nexus 5000 and 7000 have strict limitations on the length of an FCoE link as there
is not much buffer space available on those platforms.
On the Nexus 5500 where we are configuring these values the maximum supported link-size is
up to 3000 meters or 3 kilometers. We need to configure the buffer size, the buffer fill
rate at which it should start sending Pause frames, because this needs to be done before the
buffer is full and finally the buffer rate at which it can send a frame so traffic may be sent again.
We need to perform this configuration on both SW2 and SW3. Within the documentation there
is an example configuration for 3000 meter links. When we divide this by 3 and multiply by 2,
we have the buffer sizes for 2000 meter links. Which are the values we are going to use.
377
3000 meter values:
Buffer size: 152000 bytes
Pause Threshold: 103360 bytes
Resume Threshold: 83520 bytes
When we divide these by 3 and multiply by 2 we get the values for 2000 meter links.
3000 meter values:
Buffer size: 101333 bytes
Pause Threshold: 68907 bytes
Resume Threshold: 55680 bytes
By default the Nexus 5000 switches are enabled to support FCoE links of up to 300 meters,
which should be enough within any datacenter.
Now on to the configuration of SW2 and SW3.
SW2
SW2# show policy-map type network-qos

===============================
pause no-drop
mtu 2158
mtu 1500
SW2# conf t
We can not change the default policy-map, so we need to create a new policy-map of which we
base our configuration from the current default.
Then we apply our special buffer sizes.
SW2
SW2(config)# policy-map type network-qos 2000m-nq-policy
SW2(config-pmap-nq)# class type network-qos class-fcoe
SW2(config-pmap-nq-c)# pause no-drop
SW2(config-pmap-nq-c)# mtu 2158
SW2(config-pmap-nq-c)# pause no-drop buffer-size 101333 ?
pause-threshold Buffer limit for pausing in bytes
378

SW2(config-pmap-nq-c)# pause no-drop buffer-size 101333 pause-threshold 68907
resume-threshold 83520
SW2(config-pmap-nq-c)# system qos
SW2(config-sys-qos)# service-policy type network-qos 2000m-nq-policy
SW2(config-sys-qos)# exit
Now we verify our changes in configuration!

SW2(config)# show policy-map type network-qos

===============================
policy-map type network-qos 2000m-nq-policy
pause no-drop buffer-size 101333 pause-threshold 58880 resume-threshold 38400
mtu 2158
mtu 1500
pause no-drop
mtu 2158
mtu 1500
SW2(config)#
SW2(config)# show policy-map interface
Global statistics status :
enabled
port-channel3
Service-policy (qos) input:
default-in-policy
disabled
Class-map (qos):
Match: cos 3
set qos-group 1
Class-map (qos):
Match: any
set qos-group 0
Service-policy (queuing) input:

default-in-policy
disabled
Match: qos-group 1
Match: qos-group 0
Service-policy (queuing) output:
default-out-policy
disabled
Match: qos-group 1
379
Match: qos-group 0
SW2(config)# show queuing int
Ethernet1/1 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0
WRR
50
1
WRR
50
RX Queuing
qos-group 0
drop-type: drop, xon: 0, xoff: 1366
Statistics:
: 883236801
: 859564057
: 23672744
: 229759946
: 0
qos-group 1
drop-type: no-drop, xon: 690, xoff: 83
Statistics:
: 0
: 0
: 0
: 0
: 0
Total Multicast crossbar statistics:
Mcast pkts received from the cross-bar
: 16301864
380
We see that the policy-map is correctly generated and now applied to all interfaces, because
we see different queue sizes and other xon/xoff values!
We successfully completed task 4!
Task 5: N-Port Virtualization (NPV) and N-Port ID Virtualization (NPIV)
This next task is about a technology introduced to simplify the Fibre Channel networks in the
access. The switches all need to carry a lot of information and need to maintain a strict routing
protocol and zoning database.
This introduces a lot of control-plane complexity in all the Fibre Channel switches. To ease
this the idea was to create access switches which are simple and only allow devices to come
online and send traffic out over 1 or more uplinks to the Fibre Channel core network.
These switches can also operate independently, but are usually not powerful enough to be part
of a large Fibre Channel network.
381
By default when you configure an F-port where an N-port is connected to. The MDS switch
only allows up to 1 device to log-in to it. Except for Fabric Loop (FL) ports, where multiple
devices in the loop can log-in. Except that this last technology is very different.
We want normal F-ports to log-in multiple times to a single link on a smart MDS switch in the
core network.
This is where we start using N-Port Virtualization or NPV. This technology allows
devices to log-in to a fabric and then translates this request up to a core switch on a dedicated
uplink connection.
The core switch also requires some special configuration. The connections down to the
access switch (or edge switch) are just default F-ports, but a feature called N-port ID
Virtualization (NPIV) needs to be enabled to accept the translated log-ins from the
access switch in to the core switch.
The NPIV feature on the core switch enables the fact that it allows for handing out multiple
FCIDs to a single F-port.
This task and the next one will be using a little different topology to support the NPV features.
NPV requires a reboot of the device after which it can no longer be used as a normal MDS
switch. The chances of seeing this technology used on the CCIE Data Center lab is small,
because it would cost a whole MDS switch just for this single feature.
The drawing below is a subset of Drawing 3, which the topology we are using for this task.
The JBODs are the multiple hosts that want to connect to MDS1 where MDS2 should handle
their logins.
The terminology in NPV environments is that the switch where devices are connected is called
the edge switch and where the log-ins are handles is the core switch.
382
The argument in this task that the fabric should support more than 239 switches per VSAN
comes from the fact that you can not have more than 239 Domain IDs handed out in your
Fibre Channel Fabric. This is due to the fact that this is a single byte number, with some
reservations.
The NPV edge switch does not consume a FC Domain ID as its not really participating in the
Fabric. Which introduces the possibility of having over 239 switches in the Fabric (VSAN).
We should first enable the ISL links again between the switches.
MDS1
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
MDS1(config-if)#
interface fc1/1
no shutdown
interface fc1/2
no shutdown
interface fc1/3
no shutdown
interface fc1/4
no shutdown
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
interface fc1/1
no shutdown
interface fc1/2
no shutdown
interface fc1/3
no shutdown
interface fc1/4
no shutdown
After that we enable the NPIV feature on the core switch, so it allows handing out multiple
FCIDs on a single interface.
MDS2
MDS2(config-if)# feature npiv
Next we should make MDS1 the edge switch. When you enable NPV it automatically does a
write erase of the whole switch and performs a reboot. This is a very impacting command!
MDS1
MDS1(config)# feature npv
Verify that boot variables are set and the changes are saved. Changing to npv mode
erases the current configuration and reboots the switch in npv mode. Do you want to
continue? (y/n):y
>> MDS-Bootloader-01.00.19 (Feb
1 2010 - 15:13:26), Build: 01.00.19
PowerPC
CPU:
8548E, Version: 2.1, (SVR:0x80390021)
Core:
E500, Version: 2.2, (PVR:0x80210022)
Clocks: CPU:1333 MHz, CCB: 533 MHz,
383

DDR: 266 MHz, LBC: 33 MHz
D-cache 32 kB enabled
I-cache 32 kB enabled
INFO: Booting off primary flash.
I2C:
ready
DRAM: DDR: ddrioovcr 0xd0000000
Total SDRAM memory is 1024 MB
40000000
INFO: SDRAM tests PASSED.
DRAM: ECC initialization in progress...Done.
done.
INFO: Board rev = 6 type = 4 index 9037
L2 cache 512KB: enabled
IDE:
Bus 0: OK
Device 0: Model: SILICONSYSTEMS UDMA 1GB-4673 Firm: 3.38 Ser#:
CC33521774500002550
Type: Hard Disk
Capacity: 977.4 MB = 0.9 GB (2001888 x 512)
L1:
Booting bootflash:/kickstart-latest ...

...................
Automatic boot of image at addr 0x00000000 ...
Starting kernel...
Entered kgdb_console_init:1975
INIT: version 2.85 booting
Checking all filesystems....r. done.
Setting kernel variables done.
Setting the System Clock using the Hardware Clock as reference...System Clock set.
Local time: Thu Nov 8 14:07:31 UTC 2012
Loading system software
Uncompressing system image: bootflash:/system-latest
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
n_port virtualizer mode.
--------------------------------------------------------------INIT: Entering runlevel: 3
Starting NFS servers: nfsd mountd.
2012 Nov 8 14:08:08 Nov 8 14:08:08 %KERN-2-SYSTEM_MSG: Starting kernel... kernel
2012 Nov 8 14:08:09 Nov 8 14:08:08 %KERN-1-SYSTEM_MSG: Entered
kgdb_console_init:1975 - kernel
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_FAIL: Power supply 1 failed or shut down
(Serial number QCS1427K1CM)
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_OK: Power supply 2 ok (Serial number
QCS1427K1NK)
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-PS_FANOK: Fan in Power supply 2 ok
2012 Nov 8 14:08:39 MDS1 %PLATFORM-2-FAN_OK: Fan module ok
[########################################] 100%

MDS1 login: admin
Password:
384

MDS1#
Next we configure the interfaces to support the topology. We will be using all the interfaces
that we have without bundling them. Traffic is able to load balance across the available uplinks
towards the core switch.
MDS2
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
MDS2(config-if)#
interface fc1/1
switchport mode
interface fc1/2
switchport mode
interface fc1/3
switchport mode
interface fc1/4
switchport mode
f
f
f
f
Default F ports are used for connecting core to edge switches in the NPV topology.
We then ensure that the interfaces are placed into VSAN 10 as that is what the task states.
MDS2
MDS2(config-if)# vsan
database
vsan 10 interface
vsan 10 interface
vsan 10 interface
vsan 10 interface
fc1/1
fc1/2
fc1/3
fc1/4
Now that we configured our core switch to support our topology we focus on the NPV edge
switch MDS1 again.
We configure the uplinks to the core switch to be NPV uplinks.
MDS1
MDS1(config)# int fc1/1-4
MDS1(config-if)# sw mode np
MDS1(config-if)#
Once this is enabled the interfaces come only on the core switch and the NPV switch performs
an FLOGI into the core.
MDS2(config-vsan-db)# do sh int fc1/1
fc1/1 is up
Port WWN is 20:05:00:05:9b:77:0d:c0
Admin port mode is F, trunk mode is on
385

Port mode is F, FCID is 0x1e0020
Port vsan is 10
Speed is 4 Gbps
Rate mode is shared
Interface last changed at Fri Nov 9 00:03:29 2012
MDS1(config-vsan-db)# do sh flogi data

-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------fc1/1
10
0x1e0020 20:05:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1
fc1/2
10
0x1e0000 20:06:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1
fc1/3
10
0x1e0020 20:03:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1
fc1/4
10
0x1e0000 20:04:54:7f:ee:18:07:a0 20:01:54:7f:ee:18:07:a1
MDS1(config-vsan-db)# do sh fcns data
VSAN 10:
-------------------------------------------------------------------------FCID
TYPE PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x1e0000
N
20:06:54:7f:ee:18:07:a0 (Cisco)
npv
0x1e0020
N
20:05:54:7f:ee:18:07:a0 (Cisco)
npv
0x1e0040
N
20:04:54:7f:ee:18:07:a0 (Cisco)
npv
0x1e0060
N
20:03:54:7f:ee:18:07:a0 (Cisco)
npv
Now the switch is ready to receive traffic and log-ins! Lets verify on the NPV switch.
MDS1(config-if)# show npv status
npiv is disabled
disruptive load balancing is disabled
External Interfaces:
====================
Interface: fc1/5, State: Failed(Mismatch in VSAN for this upstream port)
Interface: fc1/6, State: Failed(Mismatch in VSAN for this upstream port)
386
Number of External Interfaces: 2

Server Interfaces:
==================
Number of Server Interfaces: 0
We see a mismatch in VSAN configuration. The NPV switch is still VSAN aware and you are even
able to configure multiple VSANs on the NPV switch when this would be necessary.
Therefore we also need to do our VSAN configuration on the NPV switch.
MDS1
MDS1(config-if)# vsan
database
vsan 10
vsan 10 interface
vsan 10 interface
vsan 10 interface
vsan 10 interface
fc1/1
fc1/2
fc1/3
fc1/4
After configuring the VSAN database the status looks fine and the NPV switch is now really
ready to accept logins and traffic from end hosts.
MDS1(config-vsan-db)# show npv status
npiv is disabled
====================
Interface: fc1/1,
Interface: fc1/2,
Interface: fc1/3,
Interface: fc1/4,
VSAN:
VSAN:
VSAN:
VSAN:
10,
10,
10,
10,
FCID:
FCID:
FCID:
FCID:
0x1e0020,
0x1e0000,
0x1e0040,
0x1e0060,
State:
State:
State:
State:
Up
Up
Up
Up

Server Interfaces:
==================
Now its time to configure our JBODs to be logging in to the NPV switch.
MDS1
MDS1(config-if)# switchport mode f
MDS1(config-if)# switchport mode f
MDS1(config-vsan-db)# vsan 10 interface fc1/5
MDS1(config-vsan-db)# vsan 10 interface fc1/6
387
When you would forget to add the interfaces to the VSAN database, the ports will not come
online and will give you this message.
MDS1(config)# show int fc1/5
fc1/5 is down (NPV upstream port not available)
Now when we verify our Core switch we see FLOGIs being performed against the switch, just
like we would expect from any other end host system.
MDS1(config-vsan-db)# show flogi data
-------------------------------------------------------------------------------INTERFACE
VSAN
FCID
PORT NAME
NODE NAME
-------------------------------------------------------------------------------fc1/1
10
0x1e0020 20:05:54:7f:ee:18:07:a0 20:0a:54:7f:ee:18:07:a1
fc1/1
10
0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54
fc1/2
10
0x1e0000 20:06:54:7f:ee:18:07:a0 20:0a:54:7f:ee:18:07:a1
MDS1(config-vsan-db)# sh fcns data
VSAN 10:
-------------------------------------------------------------------------FCID
TYPE PWWN
(VENDOR)
FC4-TYPE:FEATURE
-------------------------------------------------------------------------0x1e0000
N
20:06:54:7f:ee:18:07:a0 (Cisco)
npv
0x1e0020
N
20:05:54:7f:ee:18:07:a0 (Cisco)
npv
0x1e0100
N
21:01:00:e0:8b:2a:f0:54 (Qlogic)
scsi-fcp:init
Now the zoning and all other features related to this end host are applied to the core switch
and not to the edge switch. The edge switch really is nothing different than some sort of port
extender of the network.
On the NPV switch its also visible that the hosts are logging in to the switch.
MDS1(config-if)# sh npv status
npiv is disabled
====================
Interface: fc1/1,
Interface: fc1/2,
Interface: fc1/3,
Interface: fc1/4,
VSAN:
VSAN:
VSAN:
VSAN:
10,
10,
10,
10,
FCID:
FCID:
FCID:
FCID:
0x1e0020,
0x1e0000,
0x1e0040,
0x1e0060,
State:
State:
State:
State:
Up
Up
Up
Up

Server Interfaces:
==================
Interface: fc1/5, VSAN:
Interface: fc1/6, VSAN:
10, State: Up
10, State: Up
388

MDS1(config-if)#
The next question is related to steering traffic of the NPV switch across different uplinks.
389
CCIE Data Center Lab Preparation DSG
By default traffic will be balanced across the uplinks, but only once when they come online.
There will be no second time the traffic is load balanced.
Its possible to steer traffic manually across different uplinks of the NPV switch. This is done
using Traffic-Maps. There you are able to specify which server interface needs to talk with
which uplink interface.
You can first check the current load balancing configuration with the following outputs.
MDS1(config-if)# show npv traffic-map
no traffic-map configuration found
MDS1(config-if)# show npv flogi-table
-------------------------------------------------------------------------------SERVER
EXTERNAL
INTERFACE VSAN FCID
PORT NAME
NODE NAME
INTERFAC
E
-------------------------------------------------------------------------------fc1/5
10
0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54 fc1/1
fc1/6
10
0x1e0100 21:01:00:e0:8b:2a:f0:54 20:01:00:e0:8b:2a:f0:54 fc1/1
MDS1(config-if)# show npv external-interface-usage
NPV Traffic Usage Information:
---------------------------------------Server-If
External-If
---------------------------------------fc1/5
fc1/1
fc1/6
fc1/1
---------------------------------------MDS1(config-if)#
Now we are going to alter the behavior with traffic-map configuration.

MDS1
MDS1(config)# npv traffic-map server-interface fc1/5 external-interface fc1/1
MDS1(config)# npv traffic-map server-interface fc1/6 external-interface fc1/3
Now the traffic is balanced according to the questions.

Finally we should ensure that traffic is balanced over all uplinks automatically.
This means we need to enable disruptive load balancing which might cause some traffic
loss, but does ensure an optimal load balancing across all interfaces.
MDS1
MDS1(config)# npv ?
auto-load-balance
traffic-map
Configure auto load balancing among preferred external

links
Configure NPV traffic engineering
MDS1(config)# npv auto-load-balance ?
390

disruptive
Enable disruptive auto load balancing among external links
MDS1(config)# npv auto-load-balance disruptive

Enabling this feature may flap the server interfaces whenever load is not in a
balanced state. This process may result in traffic disruption. Do you want to
proceed? (y/n): y
MDS1(config)#
Now we finished task 5!

The final task is about the same feature, but then on the Nexus 5000 platform instead of the
Cisco MDS platform.
Task 6: FCoE NPV
The final task of this chapter will be about configuring the N-Port Virtualization feature
on the Nexus 5000 switch.
This mechanism works on the exact same way as the NPV feature works for Native FC ports. Of
course the Nexus 5000 switch also supports the native NPV features as well as the FCoE
features.
The technology works by configuring a VFC port just like on Multi-Hop FCoE, but instead of
configuring it as an E-port. Its configured as an NP-port.
Be careful when you enable this, because when you enable FCoE and NPV, you will perform a
reboot of the switch and enable both of the features. Whilst in this task we only require to have
FCoE-NPV enabled, which does not require a reboot of the switch!
SW2
SW2(config)# feature fcoe-npv
Do not forget to enable NPIV on SW3 otherwise it will not accept multiple logins through a
single F-port.
SW3
SW3(config)# feature npiv
We will configure the fourth uplink to SW3 as the uplink and VSAN 20. In relation to the
previous tasks we will be using VLAN 20 for the VSAN to VLAN mapping.
SW2
SW2(config-if)# vlan 20
391

SW2(config-if)# bind interface e1/8
SW2(config-if)# switchport mode np
SW3
SW3(config-if)# bind interface e1/8
SW3(config-if)# switchport mode f
And with this final configuration we finished Chapter 7!

The next chapter will be about configuring security features on the Nexus switches!
392
Chapter 8: Security
Features
Chapter 8: Security Features is intended to let you be familiar with the Security features which are
available on the Nexus platform. You will be configuring both AAA services and other management
security as well as LAN security features like DHCP snooping and other protective features.
393
General Rules
chapter
4 hours
394
Solutions
Task 1: Port Security
In this chapter we will be configuring several security features on the Cisco Nexus series
platform. We only focus on the Nexus switches in this chapter, because we already discussed
the security features on the MDS switches.
There is a slight overlap in features that are more related to management of the devices than to
the security side of the device, but according to the blueprint they fall under security features,
which is the reason why we are going to discuss them here.
The Nexus switches are really switches, so do not expect a lot of security features on these
devices like there are on IOS routers. Things like Zone Based Firewalls or anything
related to stateful security of data plane traffic is not available.
The security on the Nexus switches is purely related to the interfaces, control-plane and a little
bit to LAN based security features like DHCP snooping.
The drawing below is the generic topology drawing of this chapter regarding the hardware that
will be used in the questions of this chapter. Drawing 1 illustrates the physical topology.
One important topic is Cisco TrustSec. This technology has been introduced on the Nexus
series switches to introduce security within the datacenter.
This technology is only available on a selected number of devices and ensures you can create a
multi-tenant environment. Traffic is identified with tags. All devices that are using this tag are
authenticated between each other and traffic that is sent between the devices is secured using
encryption, anti-replay protection and message integrity checks.
395
The second major task of this chapter is the Access Control Lists (ACLs) where you
can protect VLANs, Interfaces and Layer 3 VLAN interfaces. With ACLs you are able to protect
an entity according to the Source and/or Destination IP address, IP protocol and
Source and/or Destination TCP/UDP port number. VLANs can be protected by filtering
on MAC addresses (source and/or destination).
This initial task will be about configuring a feature which has been available on Catalyst
switches for a long time and is now introduced on the Datacenter switches as well.
The port security feature allows you to configure a maximum number of hosts that are
allowed to connect to the interface. Hosts mean in this case MAC addresses. These MAC
addresses can be auto learnt or statically configured. In a data center environment you can
easily have thousands of MAC addresses so this is not often seen in these environments.
One flexibility in the feature that has been introduced in the Nexus switches is that its possible
to configure the port-security maximum amounts per VLAN. So you are able to configure
port-security on trunk links as well and specify maximums per VLAN.
There are a lot of options with this feature so therefore there are a high number of questions in
this task.
There are limitations to this feature. Some are system limits and others are configurable.
8192 is the maximum amount of MAC addresses that can be secured by the port security
feature on an entire system
1024 is the maximum amount of MAC addresses that can be learnt on any interface
You can configure a maximum per VLAN, but this is not configured by default
The feature not only protects against too much hosts logging in to the same interface, it also
protects against the moving of MAC addresses.
This means that when a certain host is learnt or configured on an interface and it is connected
to another interface. This is seen as a violation and the host is denied access or the interface
is shutdown. This is called a MAC Move Violation.
The feature does not work for vPC interfaces as the database can not be synchronized
between vPC peer switches. This introduces violations when hosts move between member
interfaces of the vPC, therefore this command is not available on vPC interfaces. It is available
on standard port-channels, where it doesnt matter to which member interface the host is
accessed.
The first questions in this task are related to setting up the Nexus switches for configuration.
We need to configure the management network and some hostnames, because the switches
boot up with a blank configuration. This is something we did a number of times, so it shouldnt
be a surprise.
SW1
396

of the system.
Please register Cisco Nexus Family devices promptly with your
service calls. Nexus devices must be registered to receive entitled
support services.
Nexus Switch
switch# conf t
SW1(config)# interface mgmt0
SW1(config-if)# ip default-gateway 172.16.100.254
SW1(config)# ntp server 172.16.100.254
After configuring the management interface and the hostname, we are good to go. The other 2
switches are configured in a similar way.
SW2
of the system.
support services.
397

Nexus Switch
switch# conf t
SW3
of the system.
support services.
Nexus Switch
398

switch# conf t
Now we start with configuring the logical topology of this lab. This logical topology is fairly
simple as we do not require much topology to test the security features.
In this lab we will simulate a lot of hosts behind interfaces, but they are not really connected.
You will see similar questions in the CCIE Data Center lab exam.
The topology is created with standard LACP port-channels between all switches and a
number of client interfaces where simulated hosts are then connected.
Below is Drawing 2 which is used as the logical topology in this chapter.
Within this chapter we will be configuring several VLANs. When VLAN numbers are mentioned
and do not exist yet you should create them on the fly while working through the task.
We will start by configuring the port-channels between the switches. Ensure you are using
LACP as this is a requirement in the task.
SW1
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
int e3/1-2
no shut
int port-chan101
sw mode trunk
399

SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
no shut
int e3/5-6
no shut
int port-chan102
sw mode trunk
no shut
After configuring the port-channel on the first switch. It will come only, but the ports will be
in isolated state, because there is no LACP communication received back from the neighboring
switch.
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group PortType
Channel
-------------------------------------------------------------------------------101
Po101(SD)
Eth
LACP
Eth3/1(I) Eth3/2(I)
102
Po102(SD)
Eth
LACP
Eth3/5(I) Eth3/6(I)
SW1(config-if)#
This will change as soon as we start configuring the other 2 switches for port-channeling
the interfaces together.
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int e1/1-2
no shut
int port-chan101
sw mode trunk
no shut
int e1/5-6
no shut
int port-chan103
sw mode trunk
no shut
SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
int e1/1-2
no shut
int port-chan102
sw mode trunk
no shut
int e1/5-6
400

SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
no shut
int port-chan103
sw mode trunk
no shut
Now when the port-channeling is checked we see that the interfaces are in participating
state which means that the port-channels are up and running.
SW1
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group PortType
Channel
-------------------------------------------------------------------------------101
Po101(SU)
Eth
LACP
Eth3/1(P) Eth3/2(P)
102
Po102(SU)
Eth
LACP
Eth3/5(P) Eth3/6(P)
SW1(config-if)#
SW2
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group PortType
Channel
-------------------------------------------------------------------------------101
Po102(SU)
Eth
LACP
Eth1/1(P) Eth1/2(P)
103
Po103(SU)
Eth
LACP
Eth1/5(P) Eth1/6(P)
SW2(config-if)#
SW3
Flags: D - Down
s - Suspended
r - Module-removed
S - Switched
R - Routed
-------------------------------------------------------------------------------Group PortType
Channel
-------------------------------------------------------------------------------102
Po102(SU)
Eth
LACP
Eth1/1(P) Eth1/2(P)
103
Po103(SU)
Eth
LACP
Eth1/5(P) Eth1/6(P)
SW3(config-if)#
401
Now we can move on to the first task of configuring the port-security feature.
This first question states that only 10 hosts may be connected to a certain ethernet interface on
SW2.
The action that should be taken when the maximum has been reached is that the interface
should be shutdown when another host is connected. Now this final point is an easy one, as by
default the action that is taken in case of a violation of the port-security rules is to shut
down
the
interface.
There are 3 actions that can be taken when a port-security rules are violated.
Shutdown
This action means that the interface will go in to errdisable state and the interface is
completely shutdown at that point. After it is re-enabled it keeps its port-security
configuration without changing anything. Including the previously learnt MAC addresses.
This is the default mode.
Restrict
Traffic from secure MAC addresses is allowed on the interface, but traffic from any
unsecure MAC addresses is dropped and a count is kept for the dropped packets
Protect
Traffic from secure MAC addresses is still allowed, but the interface is protected as MAC
address learning is disabled right after the first unsecure MAC address is seen. This
means that new MAC addresses are no longer learnt. Traffic from previously learned safe
MAC addresses can still pass through the interface
Lets start with the configuration of the interface to protect it for 10 MAC addresses.
In this case we will statically configure the violation again, to ensure its done. Although its
a default. You will never get points taken off your score for over configuration that doesnt
harm anything related to the tasks.
SW2
SW2(config)# feature port-security
SW2(config-if)# switchport port-security
SW2(config-if)# switchpor port-security maximum ?
<1-1025> Maximum addresses 1 to 1025
SW2(config-if)# switchpor port-security maximum 10 ?
<CR>
vlan
Vlan on which the mac address should be secured
SW2(config-if)# switchpor port-security maximum 10
402

SW2(config-if)# switc port-security ?
<CR>
aging
Port-security aging commands
mac-address MAC address
maximum
Max secure addresses
violation
Security violation mode
SW2(config-if)# switc port-security violation ?
protect
Security violation protect mode
restrict Security violation restrict mode
shutdown Security violation shtudown mode
SW2(config-if)# switc port-security violation shutdown
As is shown in the show run output. The violation is not configured in the configuration,
because its a default.
Do not forget to enable the port-security feature by entering switchport portsecurity. Without this command, the other commands related to it, will show up in the
configuration, but the feature is not enabled on this interface and will not work.
SW2(config-if)# do sh run int e1/11
!Time: Sat Oct 27 05:08:14 2012
version 5.1(3)N1(1)
switchport port-security
switchport port-security maximum 10
With the following output you are able to check if its correct what you configured and what the
current status of the connection is.
SW2(config-if)# do sh port-security
Total Secured Mac Addresses in System (excluding one mac per port)
Max Addresses limit in System (excluding one mac per port) : 8192
: 0
---------------------------------------------------------------------------Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action

(Count)
(Count)
(Count)
---------------------------------------------------------------------------Ethernet1/11
10
0
0
Shutdown
============================================================================
SW2(config-if)#
403
The next question states that the entries that are learnt through this interface should time-out
after a certain time. By default when MAC addresses are learnt through the interface they are
kept in the database forever, as aging is disabled.
You can change this behavior by configuring aging. This can be done in two ways.
Absolute
The absolute aging removes a MAC address after a certain amount of time. After the
MAC address has been learnt, the switch starts the timer and after the configured
amount of time, it deletes the entry from the database.
Inactivity
After the last packet received from the MAC address. The switch starts a timer, when
after it expires the MAC address is deleted from the database. When a new packet is
received
from
the
host,
the
timer
is
reset
again.
The question states that the MAC address entry should be removed after inactivity for 6
minutes. This means we are going to use this option for aging under the interface.
SW2
SW2(config-if)# switchport port-sec aging ?
time Port-security aging time
type Type of timers
SW2(config-if)# switchport port-sec aging type ?
absolute
Absolute Timer
inactivity Inactivity Timer
SW2(config-if)# switchport port-sec aging type inactivity
SW2(config-if)# switchport port-sec aging time ?
<1-1440> Aging time in minutes. Enter a value between 1 and 1440
SW2(config-if)# switchport port-sec aging time 6
After configuring the aging, you will see it is applied to the interface.
SW2(config-if)# sh port-sec int e1/11
Configured Port Security
: Enabled
Opertional Port Security
: Enabled
Port Status
: Secure Down
Violation Mode
: Shutdown
Aging Time
: 6 mins
Aging Type
: Inactivity
Maximum MAC Addresses
: 10
Total MAC Addresses
: 0
Configured MAC Addresses
: 0
Sticky MAC Addresses
: 0
Security violation count
: 0
SW2(config-if)#
404
The next question asks to configure port-security on another interface with statically
configured MAC addresses. As soon as you configure the static MAC addresses you should not
forget to also put on the maximum number of MAC addresses. When you do not configure the
maximum amount, the interface will keep learning MAC addresses indefinitely.
SW3
SW3(config)# feature port-security
SW3(config-if)# switchport port-security
SW3(config-if)# switchport port-security maximum 5
SW3(config-if)# switchport port-security mac-address ?
E.E.E
48 bit mac address format HHHH.HHHH.HHHH
EE-EE-EE-EE-EE-EE 48 bit mac address format HHHH.HHHH.HHHH
EE:EE:EE:EE:EE:EE 48 bit mac address format HHHH.HHHH.HHHH
EEEE.EEEE.EEEE
48 bit mac address format HHHH.HHHH.HHHH
sticky
Sticky MAC address
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
switchport
switchport
switchport
switchport
switchport
port-security
port-security
port-security
port-security
port-security
mac-address
mac-address
mac-address
mac-address
mac-address
(Option
(Option
(Option
(Option
1)
2)
3)
4)
0010.4431.a1b3
1022.a0f5.b3de
0011.99ff.22aa
5581.a09a.b00c
ba01.dad3.c0ff
Do not forget to enable the port-security feature on the interface itself, otherwise it will
not work.
After configuring the 5 MAC addresses statically we verify that configuration is applied.
SW3(config-if)# show port-security
: 4

(Count)
(Count)
(Count)
---------------------------------------------------------------------------Ethernet1/11
5
5
0
Shutdown
============================================================================
SW3(config-if)# show port-security int e1/11
: Enabled
: Enabled
Port Status
: Secure Down
Violation Mode
: Shutdown
Aging Time
: 0 mins
Aging Type
: Absolute
: 5
Total MAC Addresses
: 5
: 5
: 0
: 0
405
You can see that the configured MAC addresses are equal to the maximum number of allowed
MAC addresses on this interface. Meaning that only these addresses are allowed on the
interface and no other.
Then we are asked to configure an action after violating packets are received. As previously
explained. This means that we will be using the restrict action, as this is the only action
that will keep the interface up, but keeps a packet count of all violating packets. Which is what
the question asks for.
SW3
SW3(config-if)# switchport port-security violation restrict
SW3(config-if)#
Next up is the configuration of the port-channel between SW2 and SW3. We should only allow
up to 100 MAC addresses on this interface before denying access to new ones and protecting
the interface by stopping the MAC address learning feature.
This means that we will be using the protect action after a violation of the port-security
rules.
When you read ahead on the questions, you see that the interface should automatically
remember all MAC addresses that are received. This feature basically means that we want all
MAC addresses received on the port-channel to be statically configured, except we dont
want to configure them statically.
This feature is called sticky. When a MAC address is learnt and this feature is enabled, it is
automatically saved into the port-security database and never aged out.
We start with the configuration on the port-channel.
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
interface po102
switchport port-sec max 100
switchport port-sec violation protect
Dont forget to perform the configuration on both switches.

SW3
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
SW3(config-if)#
interface po102
switchport port-sec max 100
switchport port-sec violation protect
406
Now we configured the basic security, but we also want the feature enabled to save the MAC
addresses automatically into the port-security database.
SW2
SW2(config-if)# switchport port-security mac-address sticky
SW3
SW3(config-if)# switchport port-security mac-address sticky
Now after the configuration we see that the maximum is correctly configured and that the
interface is already learning port-security entries.
SW3(config-if)# do sh port-security
: 4

(Count)
(Count)
(Count)
---------------------------------------------------------------------------port-channel102
100
1
0
Protect
Ethernet1/11
5
5
0
Restrict
============================================================================
SW3(config-if)# sh port-security int po102
: Enabled
: Enabled
Port Status
: Secure UP
Violation Mode
: Protect
Aging Time
: 0 mins
Aging Type
: Absolute
: 100
Total MAC Addresses
: 2
: 0
: 2
: 0
SW3(config-if)#
Its possible to check which MAC addresses are learnt in the database. The entries are placed
into the running configuration. This means that they are not automatically saved after
reboots. This only happens after you save the configuration manually on the box.
SW3(config-if)# show port-security address
: 5
---------------------------------------------------------------------Secure Mac Address Table

---------------------------------------------------------------------Vlan
Mac Address
Type
Remaining Remotely Remotely Ports
age
learnt
aged
(mins)
out
------------------------------ -------1
BA01.DAD3.C0FF
STATIC 0
No
No
Ethernet1/11
407

1
CCEF.48CE.EBEE
STICKY 0
No
No
port-channel102
1
0010.4431.A1B3
STATIC 0
No
No
Ethernet1/11
1
0011.99FF.22AA
STATIC 0
No
No
Ethernet1/11
1
00C0.DD14.8D3C
STICKY 0
No
No
port-channel102
1
1022.A0F5.B3DE
STATIC 0
No
No
Ethernet1/11
1
5581.A09A.B00C
STATIC 0
No
No
Ethernet1/11
======================================================================
SW3(config-if)#
Finally we should ensure that the maximum of 100 MAC addresses is divided among a few
VLANs. The VLANS should get an even share, except for VLAN 10, which should get 66% of the
maximum (or 2/3).
Lets create the VLANs and configure this maximum on both switches.
VLAN 10 will get 66 of the 100 MAC addresses. Which leaves 33 for even distribution among
the remaining 3 VLANS. This means a total of 11 MAC addresses per VLAN.
SW2
SW2(config-if)# vlan 10,11,12,13
SW2(config)# int port-channel102
SW2(config-if)# switchport port-sec
ERROR: Maximum value on the port is
SW2(config-if)#
maximum 66 vlan 10
maximum 11 vlan 11
maximum 11 vlan 12
maximum 11 vlan 13
less than the vlan value
maximum 10 vlan 13
Now because we cant divide the number completely in whole numbers. We get an error
message that we configure too much MAC addresses per VLAN.
We solve this by giving the last VLAN one address less, as we cant up the total amount of MAC
addresses, but we are still dividing the amount evenly as possible.
SW2(config-if)# show run port-security
!Command: show running-config port-security
!Time: Sat Oct 27 07:10:42 2012
version 5.1(3)N1(1)
feature port-security
interface port-channel102
maximum 100
violation protect
mac-address sticky
maximum 66 vlan 10
maximum 11 vlan 11
maximum 11 vlan 12
maximum 10 vlan 13
408

switchport
switchport
switchport
switchport
port-security
port-security aging type inactivity
port-security aging time 6
port-security maximum 10
SW2(config-if)#
After the configuration we verify the configuration and check all our previous configuration.
The last 3 questions of this task are about the security of an interface on SW3. To simulate a
router we configure an interface on SW2 as a routed interface.
When we configure the SVI interface on SW3 for VLAN 100 and configure Ethernet1/7 as a
trunk interface. We should already be able to ping each other.
Do this before continuing with the port-security task so you are sure the Layer 3 path is
working correctly.
SW2
SW2(config-if)# do sh run int e1/7
!Time: Sat Oct 27 07:16:29 2012
version 5.1(3)N1(1)
no switchport
ip address 198.18.100.1/24
SW2(config-if)#
SW3
SW3(config-if)# switchport mode access
SW3(config-if)# switchport access vlan 100
SW3(config-if)#
409
After the configuration of the Layer 3 routed interface on SW2 and the access port on SW3
where we are able to configure the port-security at a later stage. We should now be able
to ping the other end.
It might happen that this doesnt work from the perspective of SW3.
SW3(config)# do ping 198.18.100.1
PING 198.18.100.1 (198.18.100.1): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
This might be related to the missing of a trunk allowed list on the port-channel between
the switches. This can cause it that the ping does not work.
SW3
SW3(config)# int po102
SW3(config-if)# sw trunk allowed vlan 10-13
After placing a trunk allowed list on the port-channel you are sure that the ping will work in
all cases.
SW3(config)# ping 198.18.100.1
PING 198.18.100.1 (198.18.100.1): 56 data bytes
ms
ms
ms
ms
ms
SW3(config)#
Now we can continue with configuring the port-security question, the final question of this
task.
We should configure port-security for a specific MAC address on the interface of SW3. The
problem is that SW2 has a different MAC address on its physical interface. So when we configure
this. The interface will go in errdisable state.
Lets configure the port-security feature as the question is expecting it.
SW3
SW3(config-if)# switchport port-secu
SW3(config-if)# switchport port-sec max 1
SW3(config-if)# switchport port-sec mac-address 1234.5678.abcd
410

SW3(config-if)#
Now we know for sure that the interface will be disabled, because the interface on SW2 is not
using this MAC address as its source.
SW3(config)# show port-sec
Port Status
Violation Mode
Aging Time
Aging Type
Total MAC Addresses
SW3(config)# show port-sec
int e1/17
: Enabled
: Enabled
: Secure Down
: Shutdown
: 0 mins
: Absolute
: 1
: 1
: 1
: 0
: 0
: 4

(Count)
(Count)
(Count)
---------------------------------------------------------------------------port-channel102
100
0
0
Protect
Ethernet1/11
5
5
0
Restrict
Ethernet1/17
1
1
0
Shutdown
============================================================================
SW3(config)# show int e1/17
Ethernet1/17 is down
To ensure that the port-security feature works like expected, we need to change the MAC
address of the SW2 interface. Then the interface comes back online and the port-security
feature learns the correct MAC address.
SW2
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
SW2(config-if)#
int e1/7
shutdown
mac-address 1234.5678.abcd
no shutdown
Lets verify the port-security feature to verify we completed this task.

SW3(config)# show port-sec address
: 4
---------------------------------------------------------------------Secure Mac Address Table

---------------------------------------------------------------------Vlan
Mac Address
Type
Remaining Remotely Remotely Ports
age
learnt
aged
(mins)
out
------------------------------ -------1
BA01.DAD3.C0FF
STATIC 0
No
No
Ethernet1/11
1
0010.4431.A1B3
STATIC 0
No
No
Ethernet1/11
411

100
1234.5678.ABCD
STATIC 0
No
No
Ethernet1/17
1
0011.99FF.22AA
STATIC 0
No
No
Ethernet1/11
1
1022.A0F5.B3DE
STATIC 0
No
No
Ethernet1/11
1
5581.A09A.B00C
STATIC 0
No
No
Ethernet1/11
======================================================================
SW3(config)#
Now we finished Task 1 and we will continue with other various security features.
Task 2: DHCP Snooping, DAI, IP Source Guard
The features discussed in this chapter are all used in conjunction with each other. They all use
the same database to perform there function.
It all comes down that we want to secure the switch against Layer 2 and Layer 3 spoofing
attacks. Because most hosts in a network are using DHCP to get their IP address its very easy to
create a Layer 2 to Layer 3 mapping on the switch to verify a correctness of the traffic
coming in.
When you are enabling DHCP snooping on a switch, it means that by default all interfaces are
seen as untrusted and the only packets that are allowed in are DHCP REQUEST packets from
hosts. You can then enable certain interfaces to become trusted, which means that you are
allowing DHCP servers to be connected through those interfaces.
Be aware when using this in a production network that the maximum number of bindings that
can be kept on a Nexus switch is 2000.
One other option what the DHCP Snooping feature is able to do, except for securing the
switch. Is that it can insert an Option 82 field in the DHCP packet. This field is used to identify
the user which is accessing the DHCP server. Based on the connected VLAN and Port number
you might get another address assigned from the DHCP server. Although this is usually only
seen in Service Provider environments, it can be interesting when you need this feature.
The DHCP Snooping feature needs to be enabled globally and second it needs to be enabled
on certain VLANs.
Lets
first
configure
the
DHCP
snooping
feature.
SW1
SW1(config)# feature dhcp
SW1(config)# ip dhcp snooping ?
<CR>
information DHCP Snooping information
verify
DHCP snooping verify
vlan
DHCP Snooping vlan
SW1(config)# ip dhcp snooping
SW1(config)# ip dhcp snooping vlan 50
412

SW1(config-if)# switchport mode access
SW1(config-if)# switchport access vlan 50
SW1(config-if)# ip dhcp snooping trust
The DHCP snooping feature is enabled on VLAN 50 and the interface Ethernet3/10 is used
for receiving DHCP Offer messages by the DHCP server. Dont forget to configure the port
within the VLAN as the question states that the DHCP server is directly connected to the switch.
So this needs to be an access interface in VLAN 50.
We are only required to configure SW1 as the question does not state anything about
configuring other switches for this feature as well.
The next question states that we should include the Option 82 information regarding the
VLAN and Port number into the DHCP packet.
This is done by enabling this feature. Typically you would expect this feature for DHCP Relay
configurations, but in this case its supported in DHCP snooping as well.
SW1
SW1(config)# ip dhcp snooping information option
We can verify that the features are enabled in the correct way as we want them to.
SW1(config)# show ip dhcp snooping
Switch DHCP snooping is enabled
DHCP snooping is configured on the following VLANs:
50
DHCP snooping is operational on the following VLANs:
50
Insertion of Option 82 is enabled
Verification of MAC address is enabled
DHCP snooping trust is configured on the following interfaces:
Interface
Trusted
-----------------Ethernet1/10
Yes
SW1(config)#
The next question asks to configure a verification. This verification can be performed by the
switch to ensure that the DHCP packet is not spoofed. This means that the switch checks if the
Source MAC address is equal to the MAC address specified within the DHCP packet.
SW1
SW1(config)# ip dhcp snooping verify mac-address
SW1(config)#
413
This concludes the configuration for the DHCP snooping feature. At this point the switch starts
building up a database of entries of clients that log-in through this switch.
Every time the DHCP server hands out a lease to a client, this information is stored into the
DHCP snooping database. This database now offers a perfect Layer 2 MAC Address to
Layer 3 IP address mapping on the switch.
This information is then leveraged by other features like the one we are configuring now.
Dynamic ARP Inspection (DAI) is a feature to really ensure you are safe against MAC
address spoofing attacks.
Dynamic ARP inspection takes care of a number of features.

Source MAC spoofing
The Source MAC address of an ARP request is checked against the DHCP snooping
database to ensure the request came from the correct interface
Optimal routing of packets
Because the DHCP snooping database knows where to find the ultimate destination of
the ARP request. It only sends it out on trusted interfaces or even where the
requested address is located
Checks for Valid ARP packets
The DAI process checks if ARP packets are send like they should be and verified that
there is no faulty information in the packet.
Now its also possible to create exceptions to this case. You can create an Access List and apply
that to the DAI process to ensure that ARP request sent for a specific range of IP addresses is
always allowed.
And finally logging is a very important topic. How long do you want to keep track of log
messages for this feature.
We start by configuring the initial question related to DAI. We want to ensure that VLAN 50 is
secured for ARP Spoofing, meaning we need to enable DAI on that VLAN.
There is one exception and that is that it should not perform these checks on the port-channel
interfaces going to the other switches.
Lets
configure
these
two
questions.
SW1
SW1(config)# ip arp inspection vlan 50
SW1(config)# int port-chan101
SW1(config-if)# ip arp inspection ?
trust Configure trust state
SW1(config-if)# ip arp inspection trust
414

SW1(config-if)# int port-chan102
SW1(config-if)# ip arp inspection trust
Now lets verify that it is enabled now.

SW1(config-if)# show ip arp inspec vlan 50
Source Mac Validation
: Disabled
Destination Mac Validation : Disabled
IP Address Validation
: Disabled
Vlan : 50
----------Configuration
Operation State
DHCP logging options
SW1(config-if)# show
Interface
------------Port-channel101
Port-channel102
: Enabled
: Active
: Deny
ip arp inspec interfaces
Trust State
----------Trusted
Trusted
Now we need to make an exception for this feature for the 198.18.50.0/28 subnet. As just
discussed its possible to let DAI ignore certain ARP packets that are related to a special ARP
Access List.
Which is what we will configure in this case.
SW1
SW1(config)# arp access-list VLAN50
SW1(config-arp-acl)# permit ip 198.18.28.0 0.0.0.15 mac any
SW1(config-arp-acl)# exit
SW1(config)# ip arp inspect filter VLAN50 vlan 50
Next we need to ensure that the last 50 messages are logged, both for denies and accepts.
SW1
SW1(config)#
SW1(config)# ip arp inspect ?
log-buffer Log Buffer Configuration
validate
Validate addresses
vlan
Enable/Disable ARP Inspection on vlans
SW1(config)# ip arp inspect vlan 50 logging ?
dhcp-bindings Logging of packet that match DHCP bindings
SW1(config)# ip arp inspect vlan 50 logging dhcp-bindings ?
all
Log all packets that match DHCP bindings
none
Do not log packets
permit Log DHCP Binding Permitted packets
SW1(config)# ip arp inspect vlan 50 logging dhcp-bindings all
SW1(config)# ip arp inspect log-buffer ?
entries Number of entries for log buffer
SW1(config)# ip arp inspect log-buffer entries ?
415

<1-1024>
Number of entries for log buffer
SW1(config)# ip arp inspect log-buffer entries 50
We first ensure that both permits and denies are logged, by configuring all logging.
Then we configure the maximum amount of entries. By default this is set to 32 messages.
SW1(config)# show ip arp inspect vlan 50
: Disabled
: Disabled
Vlan : 50
Operation State
SW1(config)# show ip
: Enabled
: Active
: All
arp inspect log
Syslog Buffer Size : 50

Syslog Rate
: 5 entries per 1 seconds
SW1(config)#
The final question related to Dynamic ARP inspection is a feature to extend the default
verification methods that it performs. You can enable some extended checks, like previously
mentioned. One of them is checking for a Valid IP address.
This means that the DAI feature will check for invalid or faulty IP addresses in ARP packets and
will drop these when it sees an invalid IP.
This feature is enabled with a single command.
SW1
SW1(config)# ip arp inspect validate ip
Now we finished the DAI configuration and we can check if all our settings are correctly applied
.
SW1(config)# show ip arp inspection vlan 50
: Disabled
: Enabled
Vlan : 50
Operation State
Interface
------------Ethernet1/1
: Enabled
: Active
: All
arp inspect interfaces
Trust State
----------Trusted
416

SW1(config)# show ip arp inspect statistics
Vlan : 50
----------ARP Req Forwarded =
ARP Res Forwarded =
ARP Req Dropped
=
ARP Res Dropped
=
DHCP Drops
=
DHCP Permits
=
IP Fails-ARP Req
=
IP Fails-ARP Res
=
0
0
0
0
0
0
0
0
arp inspect log
Syslog Buffer Size : 50

Syslog Rate
: 5 entries per 1 seconds
SW1(config)#
The next feature tested in this task is IP Source Guard.

This feature ensures a very strict IP spoofing security on a per interface level. The only way a
packet is allowed through it either when there is a DHCP Snooping database entry for the IP
address or there is a statically configured entry.
The feature is enabled per interface and cannot be enabled globally per VLAN. Keep in mind
that the feature is only enabled on VLANs where DHCP snooping is already enabled!
Lets start by configuring the first question.
SW1
SW1(config-if)# ip verify ?
source IP Source Guard related commands
SW1(config-if)# ip verify source ?
dhcp-snooping-vlan Vlans on which snooping is enabled
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
ip verify source dhcp-snooping-vlan

int e3/13
int e3/14
We then need to make an exception to interface Ethernet3/12 where a host with a statically
configured IP address is connected to.
SW1
SW1(config)# ip source ?
binding Static binding
SW1(config)# ip source binding ?
A.B.C.D IP address
SW1(config)# ip source binding 198.18.50.254 ?
E.E.E
MAC address (Option 1)
EE-EE-EE-EE-EE-EE MAC address (Option 2)
417

EE:EE:EE:EE:EE:EE
EEEE.EEEE.EEEE

SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e ?

vlan VLAN
SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e vlan 50 ?
interface Interface
SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e vlan 50 interface ?
ethernet
Ethernet IEEE 802.3z
port-channel Port Channel interface
SW1(config)# ip source binding 198.18.50.254 4019.a201.b04e vlan 50 interface
Ethernet3/12
SW1(config-if)# ip verify source dhcp-snooping-vlan
Do not forget to enable the IP Source Guard feature on interface Ethernet3/12 as well,
because otherwise IP spoofing attacks are still allowed trough as the interface is not
protected!
The final 2 questions are about another feature related to IP Source Guard, but only
applicable to Layer 3 interfaces.
On Layer 3 interfaces its easier to check if a spoofing is done as you can check if the source IP
address is really coming from that particular IP subnet.
When this check is done, so the traffic should come from the subnet on which it is received on.
Its called strict Unicast Reverse Path Forwarding Check (uRPF).
Another mode of this same technology is that the switch/router is able to reach the source IP
address based on its routing table. This is called loose mode.
We first need to configure the Layer 3 interface that we want to apply it to.
SW1
SW1(config-if)# feature interface-vlan
The mode which we need for our question is the Loose mode as we agree on reaching the
network in any way possible even including a default route.
We will configure the uRPF check on the VLAN 50 SVI interface.
SW1(config-if)# ip verify ?
unicast Unicast Reverse Path Forwarding
SW1(config-if)# ip verify unicast ?
source Validation of source address
SW1(config-if)# ip verify unicast source ?
reachable-via Specify reachability check to apply to the source address
418

SW1(config-if)# ip verify unicast source reachable-via ?
any Source is reachable via any interface
rx
Source is reachable via interface on which packet was received
When rx is given as the configuration command then the strict mode would be activated.
This means that traffic entering on a specific VLAN with IP subnet, it should be using a Source IP
address of this VLAN and Subnet.
SW1(config-if)# ip verify unicast source reachable-via any ?
<CR>
allow-default Loose Default Route Unicast Reverse Path Forwarding
SW1(config-if)# ip verify unicast source reachable-via any allow-default
SW1(config-if)#
Which completes Task 2. Now we can continue with Task 3 which is about configuring
Access Control Lists for both IP and MAC addresses.
Task 3: Access Control Lists
This task will explore the possibilities with Access Control Lists on the NX-OS. We will be
configuring security for a number of layer 3 and layer 2 ports and interfaces.
ACLs have been used since a very long time and has been available in IOS since many years. In
the past you configured ACLs with numbers. There were very specific reservations for these
ACL numbers. There was also a strict distinction between 2 types of ACLs, Standard and
Extended.
Within NX-OS there is no distinction between these types of ACLs and all ACLs are using the
Extended features. Which means that you can use all variables possible.
There are 2 types of ACLs that are tested in the CCIE Data Center lab. In the previous task we
already configured an ARP ACL. In this task we will be configuring IP Access Lists and MAC
Access Lists. The biggest difference between the 2 is that an IP ACL can be used at almost
any place in the switch to match certain IP traffic. MAC ACLs are used for VLAN Access
Control Lists (VACL). They offer the option to limit certain Layer 2 frames based on MAC
addresses to pass through the switch or not. IP Access Lists can only match on IP
addresses.
Be very careful when reading the questions in these tasks. The tasks can be very difficult in
wording, so read very well in which direction the traffic is flowing. You also need to be prepared
for questions about common applications, of which you should know the port numbers and the
traffic
characteristics.
419
The following types of traffic are able to be identified through MAC Access Lists:
Source and/or Destination MAC address
EtherType (Upper level protocol)
VLAN ID
802.1p bits (Class of Service)
IP (version 4) Access Lists are able to match a lot more different types of
traffic characteristics.
Source IP address
Destination IP address
Upper level protocol (TCP, UDP, EIGRP, GRE, OSPF, PIM, IP-IP, ESP, etc.)
ICMP types / codes
IGMP types
DSCP values
Source TCP/UDP port
Destination TCP/UDP port
Packet length
Fragmented packets
TCP bits (SYN, ACK, FIN, RST, etc.)
With all this variation of possible matching in the Access List, it gives you a lot of flexibility.
Be aware for lengthy Access Lists when a lot of different rules are applied.
Its also possible to create Access Lists that are connected to a time range. This means
that the Access List is only applied during a specified time range. Which could be that Internet
traffic is only allowed during business hours, or any other rule.
One important topic to think about in real life deployments is Logging. When every match and
deny is logged, this could cause a lot of traffic to the Supervisor / Control-Plane as this might
give thousands of log messages every second.
420
In our lab scenario we will not have any traffic running on the network, so we are safe to
configure any feature we like.
We start by configuring our first IP Access Control List.
By default, when you are configuring an Access Control List. There is already one rule
inserted into this list, which is a rule that denies all traffic when no other rule has been
specified.
The rule is always present at the bottom of the ACL and is never shown. So keep in mind that all
traffic that is not matched by any other rule is denied to pass through.
Its a common practice, especially in CCIE scenarios to insert a deny for all traffic at the bottom
of an ACL to ensure that you do not forget about it.
In the first question of the task its stated that we should secure VLAN 50 for traffic that should
not be there. This means we will be using an IP Access Control List.
Lets start by configuring our ACL for VLAN 50, be sure to pick names which are easily
remembered and easy to type.
The first requirement is to create an entry for a host to be allowed to the VLAN. If you read
carefully you will see that the questions are all related to traffic going TO the VLAN that we
want to secure.
One very important decision to make when creating ACLs, is in which direction the ACL needs
to be applied. The ACL can be applied in inbound and outbound direction.
Inbound
The inbound direction will match the traffic that comes IN to the interface, from the
layer 2 domain / subnet. It comes IN to the switch via this interface and there the traffic
is matched
Outbound
The outbound direction will match traffic that moves OUT of the interface onto the
Layer 2 domain / IP subnet. This matches traffic going TO hosts on the subnet connected
to the interface
For this task in this first Access List configuration we will be applying the ACL in outbound
direction.
SW1
SW1(config)# ip access-list VLAN50_OUT
We give the ACL a recognizable name.

SW1(config-acl)# permit ip host 198.18.255.100 198.18.50.0 ?
A.B.C.D Destination wildcard bits
421
The first task is to allow a single host (source address) to send traffic to the VLAN. We could
apply this rule to any destination address, but we need to be as specific as possible. So we will
configure the ACL with a wildcard mask which complies with the entire VLAN.
SW1
SW1(config-acl)# permit ip host 198.18.255.100 198.18.50.0 0.0.0.255
SW1(config-acl)#
Now the ACL is not applied to the interface yet, but we will wait for the application of the ACL
until we finished building all the rules that we are required to in this task.
Next question is that some Secure Web traffic, meaning SSL or HTTPS connections, is coming
from a certain range. We need to ensure that VLAN 50 receives this traffic.
We need to allow a /18 mask. This means in wildcard masks that 6 bits of the third octet
need to be a 1 which totals to 63.
SW1
SW1(config-acl)# permit tcp 198.18.128.0 0.0.63.255 eq 443 198.18.50.0 0.0.0.255 gt
1024
SW1(config-acl)#
422
The traffic is coming from servers, meaning that the source port will be Secure Web, as the
VLAN 50 will contain the clients that used this port as the Destination port.
The clients are using non-reserved ports, which means that they are using ports that are not
reserved for special cases (like HTTPS). The port range is all ports above 1024. Luckily NX-OS
foresees in this and there is a operator to match ports that are greater than a specific number
or range.
The next question will include a number of services that need to be allowed, but we cannot
configure them inside of the ACL. We need another option to configure applications that need
to be matched. This is found from a feature which is borrowed from the Cisco ASA firewalls. Its
now possible to combine certain equal parameters and use them in a single Access Control
Entry (ACE). This is just for ease of configuration as the switch then automatically generates
various entries based on the grouping.
These combinations are called object-groups. They can be made for both IP addresses and
for Applications (port numbers). In this case we will need both, because we can only have two
entries in the ACL to solve the question.
We cannot use one entry for the first subnet and the second entry for the second subnet,
because we have both TCP and UDP applications in this question. Its not possible to create a
TCP and a UDP match in a single entry, so therefore we will require 2 entries.
Lets start by configuring the object groups.
SW1
SW1(config)# object-group ?
ip
IP Object groups
ipv6 IPv6 Object groups
SW1(config)# object-group ip ?
address Address object group
port
IP port object group (can be used in IPv4 and IPv6 access-lists)
SW1(config)# object-group ip address ?
WORD Object-group name (Max Size 64)
SW1(config)# object-group ip address TASK3_5_IP
SW1(config-ipaddr-ogroup)# ?
<1-4294967295> Sequence number
A.B.C.D
A.B.C.D Network address of object-group member
A.B.C.D/LEN
A.B.C.D/nn Network prefix of the object-group member
host
Host address of the object-group member
no
end
Go to exec mode
exit
pop
push
where
SW1(config-ipaddr-ogroup)# 198.19.0.0/16
SW1(config-ipaddr-ogroup)# 198.18.192.0/24
SW1(config-ipaddr-ogroup)# exit
423
We first created the object-group related to the IP subnets that are matched. Instead of
wildcard masks its also possible in NX-OS to use the CIDR notation, making an ACL much easier
to read.
SW1
SW1(config)# object-group ip port TASK3_5_PORT
SW1(config-port-ogroup)# ?
eq
Match only packets on a given port number
gt
Match only packets with a greater port number
lt
Match only packets with a lower port number
neq
Match only packets not on a given port number
no
range
Match only packets in the range of port numbers
end
Go to exec mode
exit
pop
push
where
SW1(config-port-ogroup)# eq ?
<0-65535> Port number
SW1(config-port-ogroup)#
eq 80
eq 53
eq 110
eq 25
exit
Next we configure our applications. Ensure you are aware of the port numbers for these
common services. Its easy rememberable, because you will need these port numbers in reallife as well while troubleshooting your network.
In this case dont forget to add DNS to the group where TCP will be matched against, because
DNS uses both a TCP and a UDP port number.
SW1
SW1(config)# ip access-list VLAN50_OUT
SW1(config-acl)# permit tcp ?
A.B.C.D
Source network address
A.B.C.D/LEN Source network prefix
addrgroup
Source address group
any
Any source address
host
A single source host
SW1(config-acl)# permit tcp addrgroup ?
WORD Address group name
SW1(config-acl)# permit tcp addrgroup TASK3_5_IP ?
A.B.C.D
Destination network address
A.B.C.D/LEN Destination network prefix
addrgroup
Destination address group
any
Any destination address
eq
gt
host
A single destination host
lt
424

neq
portgroup
range

Src port group
SW1(config-acl)#
198.18.50.0/24 ?
<CR>
ack
capture
dscp
eq
established
fin
gt
log
lt
neq
packet-length
portgroup
precedence
psh
range
rst
syn
time-range
urg
permit tcp addrgroup TASK3_5_IP portgroup TASK3_5_PORT
Match on the ACK bit

Enable packet capture on this filter for session
Match packets with given dscp value
Match established connections
Match on the FIN bit
Log matches against this entry
Match packets based on layer 3 packet length
Dst port group
Match packets with given precedence value
Match on the PSH bit
Match on the RST bit
Match on the SYN bit
Specify a time range
Match on the URG bit
SW1(config-acl)# permit tcp addrgroup TASK3_5_IP portgroup TASK3_5_PORT

198.18.50.0/24
SW1(config-acl)# permit udp addrgroup TASK3_5_IP eq 53 198.18.50.0/24
SW1(config-acl)#
Then we finish the configuration by applying the object-groups to an Access List

Entry. The first matches TCP applications based on 2 object-groups and the second only
requires to match DNS for UDP traffic. So we are not using another object-group for that.
As stated before, also in the Access List itself you can use CIDR notation (/24) instead of
wildcard masks, making your ACL a lot easier to read.
The next task is to connect the ACL to the VLAN. We can do this in 2 ways. We have an SVI for
the VLAN on the Switch and because all this traffic is Layer 3 routes. We can connect it to the
SVI of the VLAN, which would be the normal case!
The question prohibits us from doing that, which means we will need the second option and
that is to apply this IP Access List to the VLAN like a VACL (VLAN ACL).
Normally those VACLs are used to match and deny Layer 2 traffic, but its also possible to
match based on IP ACLs within the VACL.
So we will be configuring a VACL to match our newly created IP ACL under. In this case it
doesnt really matter like with IP ACLs in which direction it is applied, as the VACL is always
applied to ALL traffic of the VLAN, so both inbound and outbound traffic is matched against it.
SW1
SW1(config)# vlan access-map VLAN50
425

SW1(config-access-map)# ?
action
Specify the action clause
match
Specify the match clause
no
statistics Enable per-entry statistics for the ACL
end
Go to exec mode
exit
pop
push
where
SW1(config-access-map)# match ?
ip
Display IP information
ipv6 Display IPv6 information
mac
MAC configuration commands
SW1(config-access-map)# match ip
ip
ipv6
SW1(config-access-map)# match ip ?
address Match an access list
SW1(config-access-map)# match ip address ?
WORD List name (Max Size 64)
SW1(config-access-map)# match ip address VLAN50_OUT
SW1(config-access-map)# action forward
SW1(config-access-map)# exit
SW1(config)# vlan filter ?
SW1(config)# vlan filter VLAN50 ?
vlan-list Specify list of VLANs to apply access control
SW1(config)# vlan filter VLAN50 vlan-list 50
SW1(config)#
Traffic is matched in both directions. The action that is chosen here is to forward traffic. All
traffic that is not allowed to pass through is not matched. So will hit the default deny anything
rule.
Now we are done with our first IP Access List.
Next up is another question where we are not sure about all information.
We need to protect a certain host of which we do not know the IP address. Therefore we can
only allow the entire VLAN traffic, but we need to protect the host. We do know on which port
it is connected.
IP Access Lists can be applied at multiple places. One of the more common points is the
VLAN interface (SVI). We just used a VACL to match an IP ACL. The last option is to
configure the IP ACL under a physical Interface, so only traffic from that interface is matched
against
it.
SW2
SW2(config)# ip access-list E1/15_IN
426

SW2(config-acl)# deny tcp 198.18.50.0/24 host 198.19.0.25 eq 143
We create an ACL where we deny traffic towards a single host. The NX-OS knows a shortcut for
this in terms of the host keyword in ACLs. Again keep attention to the direction of the traffic. In
this case its a client accessing a server, which should be denied. Meaning that its the
destination port number which is the application port.
SW2(config-acl)# permit ip any any
Finally we should only be securing against the particular host. All other traffic should still be
passed through this interface. So we need to explicitly allow this.
Next step is to apply the ACL to the interface.
SW2
SW2(config-if)# ip ?
access-group Specify access control for packets
port
Port policy
SW2(config-if)# ip port ?
access-group Specify access control for packets
SW2(config-if)# ip port access-group E1/15_IN ?
in
Inbound packets
out Outbound packets
SW2(config-if)# ip port access-group E1/15_IN in
SW2(config-if)#
As the traffic is coming from the VLAN going towards the server, we need to apply this ACL in
inbound direction.
Be very careful with using this command, as there are 2 different ones. The access-group
variant is used for Layer 3 traffic hitting the interface when it would be in a no switchport
configuration. Only when the port command is used in this case, the traffic is matched
against it while the port is in a normal switching mode.
Our next question is related to the security of the management interface.
You can apply a ACL for protecting the control-plane on 2 positions. The first is on the VTY.
This means that you are putting an ACL on the Telnet and SSH lines to protect others from
not connecting to the box via Telnet or SSH. This doesnt protect all other protocols.
When you protect the management interface (mgmt0), you will be protecting all traffic,
assuming that all management traffic is using this interface.
In this question we are looking for this application of the protection for rogue devices.
Access-list configuration is pretty standard and we should be familiar already. Because we are
only allowed to create a single entry we will need to use a special command.
427
Within the ACL you can create a range command, where all the ports in the specified range are
allowed in the entry.
As Telnet and SSH are using connecting port numbers, we can easily use this range command
for our question.
SW1
SW1(config-if)# ip access-list MGMT
SW1(config-acl)# deny tcp 192.0.2.0/24 range ?
<0-65535>
Port number
SW1(config-acl)# deny tcp 192.0.2.0/24 range 22-23 ?
^
% Invalid number at '^' marker.
SW1(config-acl)# deny tcp 192.0.2.0/24 range 22 ?
<0-65535>
Port number
SW1(config-acl)# deny tcp 192.0.2.0/24 range 22 23 ?
A.B.C.D
Destination network address
A.B.C.D/LEN Destination network prefix
addrgroup
Destination address group
any
host
A single destination host
SW1(config-acl)# deny tcp 192.0.2.0/24 any range 22 23
SW1(config-acl)#
Remember that you need to protect against Telnet and SSH sessions that are being set-up
towards the switch, which means the position of the port numbers.
Now we can apply the ACL to the interface. Now it doesnt need to be a port ACL, but a regular
ACL as the management interface is a Layer 3 interface.
SW1
SW1(config-acl)# int mgmt0
SW1(config-if)# ip ?
access-group
Specify access control for packets
address
Configure IP address on interface
arp
Configure ARP parameters
directed-broadcast IP directed-broadcast
port-unreachable
Enable sending ICMP port-unreachable
redirects
Send ICMP Redirect messages
unreachables
Enable sending ICMP unreachables (other than portunreachable)
SW1(config-if)# ip access-group MGMT in
SW1(config-if)#
Then finally dont forget to apply this rule to all the switches as they all need to be protected.
SW2
428

SW2(config-acl)#
SW3
SW3(config-acl)#
The next question is related to a special feature of the Nexus platform. It is related to SPAN
sessions, but this session is related to ACLs in conjunction with SPAN / Monitor sessions.
This feature is called ACL Capture.
SW1
SW1(config)# hardware access-list ?
capture
Configure ACL capture
lou
LOU
resource Hardware resource
update
Configure atomic/non-atomic update and default-result
SW1(config)# hardware access-list capture
ACL Capture enabled - disabling ACL logging for all VDCs
This feature first needs to be enabled globally, before it will work. With enabling of ACL
Capture. You will disable all ACL logging.
SW1(config)# monitor session 1 type acl-capture
The type command is very important for this configuration. A normal SPAN session cannot be
used.
SW1(config-acl-capture)# destination interface ethernet3/23
SW1(config-acl-capture)# no shutdown
^
For some reason, the normal command does not work in this case. You will need to use the
abbreviated version.
SW1(config-acl-capture)# no shut
SW1(config-acl-capture)# exit
We verify the configuration of the Monitor Session.

SW1(config)# show
session 1
--------------type
state
destination ports
mon ses 1
: acl-capture
: down (No operational src/dst)
: Eth3/23
429
Next step is to create the Access-List and to connect the SPAN session to the ACL.
SW1
SW1(config)# ip access-list SPAN
SW1(config-acl)# permit tcp any any ?
<CR>
ack
Match on the ACK bit
capture
dscp
Match packets with given dscp value
eq
established
Match established connections
fin
Match on the FIN bit
fragments
Check non-initial fragments
gt
log
Log matches against this entry
lt
neq
packet-length Match packets based on layer 3 packet length
portgroup
Dst port group
precedence
Match packets with given precedence value
psh
Match on the PSH bit
range
rst
Match on the RST bit
syn
Match on the SYN bit
time-range
urg
Match on the URG bit
SW1(config-acl)# permit tcp any any capture ?
session Session ID <1-48> for this session
SW1(config-acl)# permit tcp any any capture sess 1
The session is connected to the entry that you want to be copied to the SPAN session. Dont
forget to finish the configuration with a permit for all other traffic. As this ACL is just applied to
an interface. It would break the other traffic moving through the interface.
Finally the ACL should be applied to the interface. Again pay attention to if this is a Layer 2 or a
Layer 3 interface.
SW1
SW1(config-acl)# interface e3/22
SW1(config-if)# ip port access-group SPAN in
SW1(config-if)#
430
The next question is related to Layer 2 security. Which means that we should create a MAC
Access List for VLAN 50.
Pay close attention to how the question is formulated. The access-list should only match
traffic from the server farm. Meaning that the mentioned MAC addresses are used as source
addresses for the access list.
The MAC access-list is created in a very similar way as that IP ACLs are created.
SW1
SW1(config)# mac access-list VLAN50
SW1(config-mac-acl)# ?
capture
deny
Specify packets to reject
no
permit
Specify packets to forward
remark
Access list entry comment
statistics
Enable per-entry statistics for the ACL
end
Go to exec mode
exit
pop
push
where
SW1(config-mac-acl)#
E.E.E
EE-EE-EE-EE-EE-EE
EE:EE:EE:EE:EE:EE
EEEE.EEEE.EEEE
any
permit ?
Source MAC
Source MAC
Source MAC
Source MAC
Any source
E.E.E
EE-EE-EE-EE-EE-EE
EE:EE:EE:EE:EE:EE
EEEE.EEEE.EEEE
permit
Source
Source
Source
Source
address
address
address
address
address
(Option
(Option
(Option
(Option
1)
2)
3)
4)
0bad.c0ff.ee00 ?
wildcard bits (Option
1)
2)
3)
4)
Wildcard masks work exactly the same as with IP Access Lists. In this case we chose a
relatively easy mask.

SW1
E.E.E
EE-EE-EE-EE-EE-EE
EE:EE:EE:EE:EE:EE
EEEE.EEEE.EEEE
any
permit 0bad.c0ff.ee00 0000.0000.00ff ?

Destination MAC address (Option 1)
SW1(config-mac-acl)# permit 0bad.c0ff.ee00 0000.0000.00ff any ?

<CR>
<0x0-0xffff> MAC protocol number
aarp
Appletalk AARP
appletalk
Appletalk
capture
cos
CoS value
431

decnet-iv
diagnostic
etype-6000
etype-8042
ip
lat
lavc-sca
mop-console
mop-dump
time-range
vines-echo
vlan
DECnet Phase IV
DEC Diagnostic Protocol
Ethertype 0x6000
Ethertype 0x8042
IP (Internet Protocol V4)
DEC LAT
DEC LAVC,SCA
DEC MOP Remote console
DEC MOP dump
VINES Echo
VLAN number
You can specify which traffic is allowed, but this is not necessary in our case.
SW1
SW1(config-mac-acl)# permit 0bad.c0ff.ee00 0000.0000.00ff any
SW1(config-mac-acl)# exit
Then the access-list needs to be applied to a VACL. We already created this VACL for VLAN 50.
So ensure you do not override the previous settings.
VLAN Access Maps work in a similar way as Route-Maps do. So we can create additional
entries in the map. The first will be for the IP ACL and the second for the MAC ACL.
SW1
SW1(config)# vlan access-map VLAN50 ?
<CR>
SW1(config)# vlan access-map VLAN50 20
SW1(config-access-map)# match mac ?
address Match an access list
SW1(config-access-map)# match mac address ?
SW1(config-access-map)# match mac address VLAN50
SW1(config-access-map)# action forward
SW1(config-access-map)# exit
After the second entry in the VACL is created, we can verify that it did not overrule any of the
previous configuration of the VACL.
SW1(config)# show run | sec access-map
vlan access-map VLAN50 10
match ip address VLAN50_OUT
action forward
vlan access-map VLAN50 20
match mac address VLAN50
action forward
SW1(config)#
432
The next question is that we should be gathering statistics about all entries. This is enabled
with a single command in each of the VACL sequences.
SW1
SW1(config)# vlan access-map VLAN50 10
SW1(config-access-map)# statistics
SW1(config-access-map)# vlan access-map VLAN50 20
SW1(config-access-map)# statistics
SW1(config-access-map)#
Now we can see that entries are logged per entry.

SW1(config-access-map)# show vlan access-map
Vlan access-map VLAN50 10
match ip: VLAN50_OUT
action: forward
statistics per-entry
Vlan access-map VLAN50 20
match mac: VLAN50
action: forward
statistics per-entry
The final question of this task is related to COntrol Plane Policing (CoPP). Within the
Nexus platform the CoPP is already configured, unlike in IOS. Rate-limiters / Policers
are configured for all control-plane protocols and are in working.
They are using templates according to the features that are used on the switch. By default
these are good templates and chances are that you will never have to change them.
Therefore we do not focus on tweaking these here.
However, if you are using the box strongly for Layer 2 or Layer 3 purposes you will need to
change the template, so the routing protocols are given more space over the Layer 2 control
protocols.
Default CoPP Policy (copp-system-policy-default)
Scaled Layer 2 CoPP Policy (copp-system-policy-scaled-l2)
Scaled Layer 3 CoPP Policy (copp-system-policy-scaled-l3)
Customized CoPP Policy (copp-system-policy-customized)
The question states that we should be using optimizations for Layer 3 routing and we have a
template for that! This is exactly the change that we need to perform on both SW2 and SW3.
SW2
SW2(config-access-map)# control-plane
SW2(config-cp)# service-policy input copp-system-policy-scaled-l3
SW3
SW3(config-access-map)# control-plane
SW3(config-cp)# service-policy input copp-system-policy-scaled-l3
433
To verify that the command has been executed correctly, we can use the following show
output.
SW2(config-cp)# show copp status
Last Config Operation: service-policy input copp-system-policy-scaled-l3
Last Config Operation Timestamp: 01:21:43 UTC Nov 9 2012
Last Config Operation Status: Success
Policy-map attached to the control-plane: copp-system-policy-scaled-l3
SW2(config-cp)#
Now SW2 and SW3 are using a correct CoPP policy scaled for the primary use of Layer 3 Routing
on the box.
With this final question we finished the task and we will continue with Task 4 about AAA
services.
434
Task 4: AAA services
This fourth task is focusing on Authentication, Authorization and Accounting (AAA)

services which are available on the NX-OS platform.
AAA services are used widely for authentication and proper rights management on Cisco
devices. Within NX-OS the options are widened.
AAA services are used for management options within the NX-OS device. You can administer
WHO can log-in (Authentication), WHAT the person can do (Authorization) and WHEN the
person did something (Accounting). This ensures a very granular possibility.
Now the Authorization within NX-OS is changed thoroughly. This is because NX-OS is using a
very different authorization scheme (who can access what) than classical IOS.
In Classical IOS users were given a Privilege Level. This privilege level was then connected to
certain commands that the user was able to execute once it logged in to the router or switch
running IOS. Within NX-OS this has changed to roles. A user can be given one or more roles
and connected to a role you can allow certain command sets, or allow the configuration of
certain features. When features are enabled in a role, the user is allowed to execute all
commands related to this feature including configuration and show commands.
The role based authentication (RBAC) is not discussed in this security chapter as that is
more related to management of the device. This is why this feature is discussed in the
management chapter.
The Authentication commands are the most important as they determine when a user is
allowed to log-in to a device or not. In classical IOS you had to configure either AAA services or
use the normal logging in using a simple password and enable password.
In NX-OS there is no such thing as an Enable password, but just a username and password
connected to a certain role. When you enable AAA services in classical IOS you really have to
be careful when enabling these features. As you might even get logged out of the box.
Especially in the CCIE lab exams this is very dangerous as you might need to go to the proctor
and ask for a reload of your device. Much better to prevent that from happening.
Within NX-OS this is simpler. There is no distinction between enabling the AAA services, versus
a classical mode. You are always in the same mode, making life much easier.
You can configure whether the device authenticates using the local database, which is
default, or using an AAA server.
NX-OS knows the 2 most widely used AAA protocols.
RADIUS
Based on open standards and used in all platforms from all vendors.
435
Based on UDP traffic, but including guarantees if requests were received or not.
TACACS+
Cisco proprietary protocol. However it does finds its integration in other vendor
products
Based on TCP traffic
There are not real differences in RADIUS or TACACS+, other than the type of traffic. As of
features they both offer equal features. Sometimes applications are using only RADIUS or
TACACS+, but overall they are equal in terms of features and functionality.
In the first few questions of this task we are configuring the AAA servers. We cannot use any
global configuration for that, which means we should be using groups for server
configurations. The advantage of groups over global configuration is that groups can be used
in different places in the configuration. The groups also offer failover to other AAA servers
when the primary one failed.
First we should configure the RADIUS and TACACS+ servers while doing configuration in
groups.
SW1
SW1(config)# aaa group ?
server Configure aaa server group
SW1(config)# aaa group server ?
radius RADIUS server group name
SW1(config)# aaa group server radius ?
WORD RADIUS server group name (Max Size 127)
SW1(config)# aaa group server radius RADIUS_SERVERS
SW1(config-radius)# ?
deadtime
Duration for which non-reachable server is skipped
no
server
RADIUS server name or IP address
source-interface Source interface to be used to reach radius server
use-vrf
Vrf to be used to contact servers in this group
end
Go to exec mode
exit
pop
push
where
SW1(config-radius)# server 172.16.100.201
specified RADIUS server not found, first configure it using radius-server host ...
and then retry
Now we cant configure a RADIUS server within a group before the server has been globally
configured. So we need to configure it in global configuration first.
SW1
SW1(config-radius)# exit
436

SW1(config)# radius-server host 172.16.100.201 key IPexpertAAA
SW1(config)# radius-server host 172.16.100.201 key IPexpertAAA ?
<CR>
accounting
Use for accounting
auth-port
RADIUS server's port for authentication
authentication Use for authentication
pac
Secure Radius Enable
retransmit
RADIUS server retransmit count
timeout
RADIUS server timeout period in seconds
SW1(config-radius)# use-vrf management
SW1(config-radius)# source-interface mgmt0
SW1(config)#
After configuring the global server we can configure the server inside the group. Where we
can also specify the source interface and source VRF that we want to use for RADIUS
requests.
Next we will configure the TACACS+ server configuration.
SW1
SW1(config)# taca?
^
?
^
SW1(config)# taca
^
radius RADIUS server group name
SW1(config)# aaa group server tacacs
^
SW1(config)#
By default only RADIUS configuration is enabled. For TACACS+ you need to enable the
feature first.
SW1
SW1(config)#
SW1(config)# feature tacacs+
SW1(config)#
SW1(config)# tacacs-server host 172.16.100.202 ?
<CR>
key
TACACS+ shared secret
port
TACACS+ server port
test
Paremeters to send test packets
timeout TACACS+ server timeout period in seconds
SW1(config)# tacacs-server host 172.16.100.202 key IPexpertAAA
437

radius
RADIUS server group name
tacacs+ TACACS+ server group name
SW1(config)# aaa group server tacacs+ TACACS_SERVERS
SW1(config-tacacs+)# server 172.16.100.202
SW1(config-tacacs+)# use-vrf management
SW1(config-tacacs+)# source-interface mgmt0
Now we have configured our servers both for TACACS+ and RADIUS.
As there is no specification on which switch the configuration should be applied, we need to
configure it on all switches.
SW2
SW2(config)# feature tacacs
SW3
SW3(config)# feature tacacs
The next question is related to monitoring the RADIUS servers. Within NX-OS its possible to
pro-actively monitor the reach ability of the AAA servers, which prevents timeouts while logging
in.
First we need to configure the time-out values on the groups before its declared automatically
dead.
SW1
SW1(config-radius)# deadtime ?
<0-1440> Length of time, in minutes
SW1(config-radius)# deadtime 22
438

SW1(config-tacacs+)# deadtime 22
SW1(config-tacacs+)# exit
SW2
SW3
Next we configure the monitoring. This is done by doing a simple AAA log-in to the server.
Which is why we need the username and password.
SW1
SW1(config)# radius-server ?
deadtime
Duration for which non-reachable server is skipped
directed-request Enable direct authentication requests to server
host
RADIUS server's DNS name or its IP address
key
Global RADIUS server shared secret
retransmit
Global RADIUS server retransmit count
test
Parameters to send test packets
timeout
Global RADIUS server timeout period in seconds
SW1(config)#
idle-time
password
username
radius-server test ?
Time interval for monitoring the server
User password in test packets
User name in test packets
SW1(config)# radius-server test username ipexpert password IPexpert123 idle-time 2

SW1(config)# tacacs-server test username ipexpert password IPexpert123 idle-time 2
SW1(config)#
SW2
SW3
439
By default there is a lengthy time-out value for AAA requests. Which means that users need
to wait long before they can log-in using another AAA server or perform a fall-back to the local
database.
You are also able to configure the amount of retransmits that should be done.
Therefore we will configure a short time-out value for log-in requests.
SW1
SW1(config)# radius-server timeout 2
SW1(config)# tacacs-server timeout 2
SW1(config)#
SW2
SW2(config)#
SW3
SW3(config)#
This finalizes the configuration for AAA servers on the devices.

Lets verify if our configuration was applied correctly.
SW1(config)# show radius-server
retransmission count:1
timeout value:2
deadtime value:0
source interface:any available
Global Idle Time:2
Global Test Username:ipexpert
Global Test Password:IPexpert123
total number of servers:2
following RADIUS servers are configured:
172.16.100.201:
available for authentication on port:1812
available for accounting on port:1813
RADIUS shared secret:********
SW1(config)# show tacacs-server
timeout value:2
deadtime value:0
source interface:any available
Global Idle Time:2
Global Test Username:ipexpert
Global Test Password:IPexpert123
440

total number of servers:1
following TACACS+ servers are configured:
172.16.100.202:
available on port:49
TACACS+ shared secret:********
SW1(config)#
In the next questions we will be applying the AAA configuration to the log-in process of the
switches.
Keep in mind that by default when you configure authentication, it only applies to the telnet
and SSH sessions. Console sessions are left out, as they are configured with a separate
command.
This is done, first of all to ensure you do not configure any faulty parameters causing you to be
locked out from the switch. Besides that its common that you configure the console
connection only with a local authentication, because usually you only perform console log-in,
when there is something wrong with the switch. Meaning that the chance is high that you are
also not able to reach the AAA servers anymore.
First we only want SW2 to authenticate to the RADIUS server and a fall-back to the local
database.
The local database should only be queried once the RADIUS server is dead and does not
respond. In case the RADIUS server replies with a reject message. Its not falling back to the
local database. This is because it means that the user is not allowed to log-in to the device. So a
fall-back would violate the authentication rules.
Now when the fall-back would be done. The switch would automatically use the same
username and password as wouldve been typed in. First we configure RADIUS authentication
on SW2.
SW2
SW2(config)# aaa authentication ?
dhchap Configure methods for dhchap
login
Configure methods for login
SW2(config)# aaa authentication login default ?
fallback Configure fallback behavior
group
Specify server groups
local
Use local username authentication
none
No authentication
SW2(config)# aaa authentication login ?
ascii-authentication Enable ascii authentication
chap
CHAP authentication for login
console
Configure console methods
default
Configure default methods
error-enable
Enable display of error message on login failures
mschap
MSCHAP authentication for login
mschapv2
MSCHAP V2 authentication for login
SW2(config)# aaa authentication login default ?
441

fallback
group
local
none
Configure fallback behavior

Specify server groups
Use local username authentication
No authentication
SW2(config)# aaa authentication login default group ?

WORD Server group name (Max Size 127)
SW2(config)# aaa authentication login default group RADIUS_SERVERS ?
<CR>
WORD
Server group name (Max Size 127)
none
No authentication
SW2(config)# aaa authentication login default group RADIUS_SERVERS
SW2(config)# aaa authentication login default fallback ?
error Fallback in case all AAA servers configured for remote authentication are
unreachable
(Authentication error)
SW2(config)# aaa authentication login default fallback error ?
local Fallback to local authentication
SW2(config)# aaa authentication login default fallback error local
SW2(config)#
Now when we do not configure any second method the switch will automatically fall-back to
the local username/password database and will not re-ask for a password, because its the
default. In our case we configured the fallback to be sure enough.
Next up is the console authentication which was just mentioned to be done differently from the
SSH and Telnet log-in. As there is no specification of the switch. It should be configured on all.
SW1
SW1(config)# aaa authentication login console local
SW2
SW3
Next question relates to the configuration of SW3 for authentication. This is where we need to
use a Cisco Proprietary protocol, meaning that we will use TACACS+ for this host.
SW3
SW3(config)# aaa authentication login default group TACACS_SERVERS
442
Next up is that users without a role, should not be able to log-in to the device. Users will get
roles pushed through RADIUS and TACACS+ attribute configurations.
By default when users do not get this attribute pushed. They will be configured with the
network-operator or vdc-operator role.
This can be disabled with a single command.
SW1
SW1(config)# no aaa user default-role
SW2
SW3
The next question is related to the notification for AAA servers if the servers are available.
By default users do not get to know how they have been logged in, of course for security
purposes. We can enable this feature.
SW1
SW1(config)# aaa authentication login ?
ascii-authentication Enable ascii authentication
chap
CHAP authentication for login
console
Configure console methods
default
Configure default methods
error-enable
Enable display of error message on login failures
mschap
MSCHAP authentication for login
mschapv2
MSCHAP V2 authentication for login
SW1(config)# aaa authentication login error-enable
SW2
SW3
This can be verified when logging into the switch using SSH. There its shown that the AAA
servers were not reachable for log-in and that the local database is used instead.
TerminalServer#ssh 172.16.100.1
Nexus 5000 Switch
Password:
Remote AAA servers unreachable; local authentication done
443

SW1#
The next question is related to how NX-OS stores usernames and passwords in its database.
By default it stores them with an MD5 hashed password in the database, but this can be even
stronger. By using the AES encryption algorithm.
This is known as type 6 passwords. Where MD5 hashes are known as type 5.
Before this feature can be enabled, you will need to configure a Master Key which is used for
encrypting the passwords.
There is no key specified, so you can use something you like.
SW1
SW1# key config-key ascii
New Master Key: IPexpert123
Retype Master Key: IPexpert123
Next we can enable the feature.

SW1
SW1# conf t
SW1(config)# feature password encryption aes
We should be encrypting the currently configured usernames and passwords so they are also
using the new encryption mechanism.
SW1(config)# exit
SW1# encryption re-encrypt obfuscated
Finally, dont forget to enable it on all the switches, because there was no specification given on
which switch it should be enabled.
SW2
New Master Key: IPexpert123switch2
Retype Master Key: IPexpert123switch2
444

SW2# conf t
SW2(config)# exit
As the authentication is a local decision we can use any key we want on the switch. Its even
recommended because this increases the security between the switches.
SW3
New Master Key: IPexpert123switch3
Retype Master Key: IPexpert123switch3
SW3# conf t
SW3(config)# exit
Finally we should enable accounting towards the RADIUS server to ensure that all commands
that any user typed in will be logged.
SW2
SW2(config)# aaa accounting default group RADIUS_SERVERS
As previously mentioned. NX-OS uses another mechanism than Classical IOS for giving rights to
users. This means that all AAA servers are currently configured to support the IOS privilege
levels. These are levels between 0 and 15 for determining what users are able to do on the
switches.
The NX-OS device performs a default mapping for privilege levels received through TACACS+ to
default existing roles on the devices.
Privilege
Mapped role
Some Show and Exec commands can be executed
1 to 13
Same as 0 and separately configured features using role configuration related

to privilege levels
14
vdc-admin role
15
network-admin role
445
You can also configure privilege levels within NX-OS. This needs to be enabled to configure
the privileges connected to the usernames. Its also possible to configure the privilege
levels as roles within the configuration.
To just enable backwards compatibility what we want, we just need to enable the feature.
SW3
SW3(config)# feature privilege
Next up is the forcing the users to configure strong passwords (or not).
A strong password consists out of the following characteristics.
Minimal of 8 characters long
Does not contain many consecutive characters (such as abcd)
Does not contain many repeating characters (such as aaabbb)
Does not contain dictionary words
Does not contain proper names
Contains both uppercase and lowercase characters
Contains at least 1 or more numbers
By default this strength check for passwords is enabled. When you disable and re-enable
the feature. It does not check for the current passwords to be configured with a strong
password or not. Keep this in mind in a production network.
In this case we should disable the check on SW3.
SW3
SW3(config)# no password ?
strength-check Strength check of password
SW3(config)# no password strength-check
SW3(config)#
The final question of this task is to configure a username which expires. After the expiration
date and time. The user will not be able to log-in anymore.
SW3
SW3(config)# username rick ?
<CR>
expire
Expiry date for this user account(in YYYY-MM-DD format)
keypair
Generate SSH User Keys
password
Password for the user
priv-lvl
Privilege level which the user is to be assigned to
role
Role which the user is to be assigned to
ssh-cert-dn Update cert dn
446

sshkey
Update ssh key for the user for ssh authentication
SW3(config)# username rick expire ?

WORD Expiry in YYYY-MM-DD format (Max Size 10)
SW3(config)# username rick expire 2013-12-31 ?
<CR>
password Password for the user
priv-lvl Privilege level which the user is to be assigned to
role
Role which the user is to be assigned to
SW3(config)# username rick expire 2013-12-31 password IPexpertIsAwesome role
network-admin
SW3(config)#
447
Now we finished Task 4 and we will continue with a feature not specifically mentioned on
the blueprint, but very likely to be tested as other things might be depending on it.
Besides that it is using the RADIUS configuration from this task to simulate authentications.
Task 5: 802.1X
This fifth task will be about configuring the IEEE 802.1X feature. This feature takes care of
ensuring the identity of users connecting to the network.
This is done by offering users a username/password dialog screen. Where they need to enter
there network credentials before the proper VLAN is configured on the switchport.
Without authentication users can not access the network!
This technology is actually related to LAN security and usually found in workgroup switches.
Within a datacenter environment this is usually not seen.
The code is implemented in NX-OS, but only available in the Nexus 7000. The Nexus 5000 has
no features for configuring this.
The items is not on the CCIE Data Center lab blueprint, but the chances are that it is treated in
the lab exam, because for the next task about Cisco TrustSec it is required.
Because the 802.1X feature is not fully integrated in the OS it doesnt have all the features
that it would usually have in a LAN environment.
The feature will not be a large portion of the exam, which is why we will not focus a lot on the
feature here.
First we are required to configure some basic functionality of 802.1X.
We need to be sure that on some of the interfaces of SW1 users are authenticated before they
get access to the network. They should be authenticated against the RADIUS server which we
configured in the previous task.
SW1
SW1(config)# feature dot1x
SW1(config)# aaa authentication dot1x default group RADIUS_SERVERS
SW1(config)#
First we configure the RADIUS authentication that should be used, before any of the interface
configuration is done.
SW1
SW1(config-if)# dot1x port-control auto
SW1(config-if)#
448
The interfaces are enabled with a single command. Relatively easy feature to enable, especially
because not all code is implemented on the Nexus 7000 platform. There is no such thing as
Guest VLANs or other advanced features.
The feature is purely used for authentication of the user against the network.
By default the 802.1X standard allows for a single host to be connected through the interface.
When there are multiple hosts connected through the same interface they are not allowed
access unless they authenticate.
This is disabled by default, so when we want this, we need to enable support for multiple
hosts.
SW1
SW1(config-if)# dot1x host-mode multi-host
Next we should configure a re-authentication timer. By default the reauthentication is disabled and when a host connects and authenticates it can access the
network forever. As soon as it disconnects and connects again it needs to re-authenticate
again.
When we configure re-authentication, the host will get a request to authenticate to the
switch again, to be sure that still the same user is connected to this interface.
This re-authentication can be configured globally or per interface. In our case we will
configure it at interface level, because we are not allowed to configure it globally.
SW1
SW1(config-if)# dot1x re-authentication
SW1(config-if)# dot1x timeout re-authperiod 3600
449
Re-authentication timers are given in seconds. In our case for 1 hour we need to specify
3600 seconds as re-authentication timer.
The next task is about giving access to a printer on our network which has no client for 802.1X
authentication. We still want the device to get access to the network in the correct VLAN, but
as the printer has no client software, it cannot do this by itself so we need to help in this
process.
What we can do is to use the MAC address of the client, in this case the printer, as the
authentication parameter towards RADIUS.
The RADIUS server will receive a request for authentication filled out with the clients MAC
address. We need to make sure an entry exists for this MAC address and then the port is still
authenticated.
This feature is by default disabled and can be enabled on a per interface level.
SW1
SW1(config-if)# dot1x mac-auth-bypass
The next question states that a client is allowed to re-issue authentication up to 4 times. This
means that it is allowed to re-authenticate after rejection of the RADIUS Server for another 3
times before the interface is fully shutdown.
We can also configure this parameter both globally and per interface. We will configure it per
interface as well.
SW1
SW1(config-if)# dot1x max-reauth-req 4
Finally we should ensure that all activity is logged via accounting towards the RADIUS server.
SW1
SW1(config)# dot1x radius-accounting
SW1(config)# aaa accounting dot1x default group RADIUS_SERVERS
450
With this last configuration we finished the 802.1X task. Next up is a true Data Center feature
called Cisco TrustSec.
Task 6: Cisco TrustSec
The final task of this chapter is about configuring Cisco TrustSec. This technology was
introduced within the Nexus series. It supports encryption of traffic in the data-plane. This
means it has no impact on the forwarding performance while still begin highly secure.
Besides adding encryption it also supports multi-tenancy in the network. This is because the
feature introduces a tag, called the Security Group Tag (SGT). This tag is added to the
packet, like a 802.1Q header.
Cisco TrustSec (CTS) works together with 802.1X for full security control over the entire
data center network.
The tagging requires support in hardware, so be aware of this when selecting platforms and
CTS is a requirement.
Besides encrypting traffic and tagging traffic it can also be used to selectively add switches to
the network as they are authenticated. This works almost the same as Port Security in MDS
switches and Fibre Channel links. Switches send CHAP or EAP challenges to each other to
authenticate.
All the verification can also be performed by a RADIUS server. Be aware that only Cisco ACS
supports
the
TrustSec
features
at
the
time
of
this
writing.
451
As we can not test the full features in this lab, because of limited available hardware and
software. We will focus on the authentication of the switches themselves. So a secure data
center network is created.
We first enable Cisco TrustSec and the authentication against the RADIUS server.
SW1
SW1(config)# feature dot1x
SW1(config)# feature cts
SW1(config)# cts device-id SW1 password SW1p@ssw0rd
When enabling the feature its required to have it configured with a local ID and password. We
will use the password mentioned later in this task for the other questions.
SW1
SW1(config)# aaa authentication cts default group RADIUS_SERVERS
SW1(config)# aaa authorization cts default group RADIUS_SERVERS
Do not forget to enable this on all the switches!

SW2
SW2(config)#
SW2(config)#
SW2(config)#
SW2(config)#
feature cts
cts device-id SW2 password SW2p@ssw0rd
aaa authentication cts default group RADIUS_SERVERS
aaa authorization cts default group RADIUS_SERVERS
SW3
SW3(config)#
SW3(config)#
SW3(config)#
SW3(config)#
feature cts
cts device-id SW3 password P@ssw0rdSW3
aaa authentication cts default group RADIUS_SERVERS
aaa authorization cts default group RADIUS_SERVERS
Next we should enable Cisco TrustSec (CTS) on the interfaces that we enabled 802.1X
on.
SW1
SW1(config-if)# cts dot1x
452
The enabling of CTS requires a flap of the interface before it will work. In our case we have no
hosts connected to the interface, so it doesnt really matter. In a real life network the shutdown
and no shutdown of the interface is required for the CTS feature to work properly.
The last few questions of this chapter are about another feature of Cisco TrustSec, which is
the authentication of switches to ensure your datacenter network is only consisting of switches
that you expect. Connections between switches do not come online automatically without
configuring a proper authentication.
This feature is called the SGT Exchange Protocol (SXP). It is primarily used for giving out
the Security Tags to other switches, but can therefore also be used for authenticating switches.
This is where we are going to use it for.
It requires manual configuration of the authentication keys as we cannot use RADIUS for this
task. We also cannot use the mgmt0 interface for the SXP protocol, so we need to create SVIs
which we are allowed to.
SW1
SW1(config-if)# interface po101
SW1(config-if)# sw trunk allowed vlan add 99
SW1(config)#
SW2
SW2(config)#
SW3
SW3(config)#
453
Finally we can configure the authentication parameters of the switches to ensure the SXP
protocol will run successfully on all switches.
The switches will need to be statically configured as either speaker or listener switches.
Not at the same time they should be doing both to each other.
SW1
SW1(config)#
SW1(config)#
SW1(config)#
SW1(config)#
listener
SW1(config)#
listener
feature cts
cts role-based enforcement
cts sxp enable
cts sxp connection peer 198.18.99.2 password required SW2p@ssw0rd mode
cts sxp connection peer 198.18.99.3 password required P@ssw0rdSW3 mode
SW2
SW2(config)#
SW2(config)#
SW2(config)#
speaker
SW2(config)#
speaker

cts sxp enable
cts sxp connection peer 198.18.99.3 password required P@ssw0rdSW3 mode
Dont forget to configure the speaker parameter on this switch.

SW3
SW3(config)#
SW3(config)#
SW3(config)#
speaker
SW3(config)#
listener

cts sxp enable
With this final configuration we completed the task and finished the chapter!
We prepared some configuration for the next chapter, so leave all configuration in place while
moving on to the next chapter about management features!
454
Chapter 9:
Management
Features
Chapter 9: Management Features is intended to let you be familiar with the Management features
which are available on the Nexus platform. You will be configuring Role Based Access Control (RBAC),
SNMP, Syslog, NetFlow, NTP and many more.
We highly recommend to create your own diagram at the beginning of each lab so you are able to draw
455
General Rules
chapter
4 hours
456
Solutions
Task 1: Role Based Access Control (RBAC)
This chapter is about configuring management features of the NX-OS platform. The
management features are equal to those that are available on the MDS switches as its running
the same code internally.
One of those features is also a bit related to security and that is Role Based Access
Control (RBAC).
The model of RBAC is completely different from the Privilege level style like was used in
Classical IOS. In IOS users were connected to a privilege level, which is a number between 0
and 15, on which privileges are configured. Those privileges are configured in the form
of which commands are allowed to be executed by that user.
The possibilities with this type of configuration is limited and this is changed in NX-OS. Now
roles are configured instead of privilege levels. Inside of the role configuration its still
possible to configure commands that are allowed to be executed, but they offer much more
features.
One of the possibilities of role configurations is that you are able to allow features to be
individually configured. This means you do not need to configure all the related commands in
the configuration, but can allow a whole set of configuration commands with just a single line in
the configuration by allowing a specific feature.
Per feature you are able to specify whether the user is allowed to only verify the working or
also change the configuration.
This configuration can then even be applied only to interfaces and VRFs that are configured in
the role. When a user has certain interfaces and/or VRFs specified, its not able to touch any
configuration on other interfaces and in other VRFs.
The role configuration is specific to the VDC and never shared.
There are a number of default roles pre-configured on the NX-OS devices.
457
Role
Privileges
network-admin
Complete access to all commands and configuration items
network-operator
Access to all show commands except show running-config, users can

never change any configuration
vdc-admin
Complete access to all commands and configuration items limited to a

VDC
vdc-operator
Access to all show commands except show running-config, users can

never change any configuration limited to a VDC
Its not possible to change any of the default roles.

When changes are required, you need to create new user roles for this use.
Users are not limited to a single user role, but can be a member of multiple roles. In this
case it becomes very flexible what the user is allowed to do.
When features are overlapping in a role configuration or in a configuration where a user is
connected to multiple roles. The feature or role that is applied later, so later in the chain of
configuration, that rule is applied.
This is different from the rule that usually the first or the most restrictive rule is applied.
Our configuration will begin with the configuration of certain users connected to certain roles.
Pay close attention to the questioning. Its highly advisable to read ahead and check all the
questions related to a certain role or user configuration.
You might find limitations that would otherwise mean you have to re-do some configuration
afterwards, which costs time.
In the first question its stated to configure a user with a specific role. When we try to
configure the user up front, it becomes impossible to do this. We can only configure the default
roles and the special privilege roles, because we enabled backwards compatibility with the
IOS-Style privilege levels in the previous chapter.
SW1
SW1(config)# username user1 role ?
network-admin
System configured role
network-operator System configured role
priv-0
Privilege role
priv-1
Privilege role
priv-10
Privilege role
priv-11
Privilege role
priv-12
Privilege role
priv-13
Privilege role
priv-14
Privilege role
priv-15
Privilege role
priv-2
Privilege role
458

priv-3
priv-4
priv-5
priv-6
priv-7
priv-8
priv-9
vdc-admin
vdc-operator
Privilege role
Privilege role
Privilege role
Privilege role
Privilege role
Privilege role
Privilege role
SW1(config)# username user1 role
This means we have to do our user configuration in a later stage of the configuration. We first
need to configure the role itself and the limitations.
We do not need to enable a certain feature, like with all options in NX-OS. The RBAC feature is
by default enabled and can not be disabled.
As the question states. We are not able to configure features directly under the role
configuration. In NX-OS its possible to combine features together in so-called featuregroups. These groups can then be used multiple times in different role configurations to
prevent overlapping configuration items.
SW1
SW1(config)# role feature-group ?
name Feature-group name
SW1(config)# role feature-group name ?
WORD Enter feature-group name (Max Size 32)
SW1(config)# role feature-group name USER1_FEATURES
SW1(config-role-featuregrp)# ?
feature Feature name
no
end
Go to exec mode
exit
pop
push
where
The feature names can be difficult to get to know while the CLI will not auto-complete you in
this case. It will tell you when a faulty feature is configured.
There is a list of features available through a show command which might make it a lot easier to
configure the feature names, although they are easy to guess as well!
SW1(config-role-featuregrp)# show role feature
aaa
(AAA service related commands)
arp
(ARP protocol related commands)
cdp
(Cisco Discovery Protocol related commands)
l3vm
(Layer 3 virtualization related commands)
ping
(Network reachability test commands)
snmp
(SNMP related commands)
radius
(Radius configuration and show commands)
syslog
(Syslog related commands)
tacacs
(TACACS configuration and show commands)
459

install
(Software install related commands)
license
(License related commands)
callhome
(Callhome configuration and show commands)
platform
(Platform configuration and show commands)
access-list
(IP access list related commands)
svi
(Interface VLAN related commands)
hsrp
(Hot Standby Router Protocol related commands)
igmp
(Internet Group Management Protocol related commands)
msdp
(Multicast Source Discovery Protocol related commands)
vlan
(Virtual LAN related commands)
dot1x
(DOT1X related commands)
ipfib
(IP Forwarding Information Base related commands)
eth-span
(Ethernet SPAN related commands)
router-bgp
(Border Gateway Protocol related commands)
router-rip
(Routing Information Protocol related commands)
ethanalyzer
(Ethernet Analyzer)
router-ospf
(Open Shortest Path First protocol related commands)
router-eigrp
(Enhanced Interior Gateway Routing Protocol related commands)
spanning-tree
(Spanning Tree protocol related commands)
acl
(FC ACL related commands)
sfm
(ISCSI flow related commands)
sme
(Storage Media Encryption feature related commands)
fcns
(Fibre Channel Name Server related commands)
fcsp
(Fibre Channel Security Protocol related commands)
fdmi
(FDMI related commands)
fspf
(Fabric Shortest Path First protocol related commands)
rlir
(Registered Link Incident Report related commands)
rscn
(Registered State Change Notification related commands)
span
(SPAN session relate commands)
vsan
(VSAN configuration and show commands)
wwnm
(WorldWide Name related commands)
zone
(Zone related commands)
fcanalyzer
(FC analyzer related commands)
sme-kmc-admin
(SME commands authorized to kmc admin)
sme-stg-admin
(SME commands authorized to storage admin)
sme-recovery-officer(SME commands authorized to recovery officer)
SW1(config-role-featuregrp)#
Now we know enough to perform the configuration for the role of user1.
SW1
feature
feature
feature
feature
vlan
svi
spanning-tree
hsrp
We need to configure access to FHRP protocols. In this case this only means HSRP, as thats the
only feature that we can configure. When we try to configure the other FHRPs we are not
able to perform this configuration.
SW1(config-role-featuregrp)# feature vrrp
Invalid role feature name vrrp
SW1(config-role-featuregrp)# feature glbp
Invalid role feature name glbp
SW1(config-role-featuregrp)# exit
460
Then we can start configuring the role itself with the feature group.
SW1
SW1(config)# role ?
feature-group Configure role feature-group
name
Enter the role name
SW1(config)# role name USER1_ROLE
SW1(config-role)# ?
description Add a description for the role
interface
Configure the interface policy for this role
no
rule
Enter the rule number
vlan
Configure the vlan policy for this role
vrf
Configure the vrf policy for this role
vsan
Configure the vsan policy for this role
end
Go to exec mode
exit
pop
push
where
SW1(config-role)# rule ?
<1-256> Enter the rule number
SW1(config-role)# rule 1 ?
deny
Deny rule
permit Permit rule
SW1(config-role)# rule 1 permit ?
command
Command line
read
Read access
read-write Read and write access
SW1(config-role)# rule 1 permit read-write ?
<CR>
feature
Feature name
feature-group Feature group name
SW1(config-role)# rule 1 permit read-write feature-group ?
WORD Enter the access entity name (Max Size 32)
SW1(config-role)# rule 1 permit read-write feature-group USER1_FEATURES
The configuration is done by creating rules in the role. There you are able to specify the
read-only or read-write functionality and then the feature or feature-group that
should get access when connected to a username.
Next up is that the user should be limited to only configure a few interfaces.
This is done by first denying access to all interfaces and later specifying which interfaces should
have access.
SW1
SW1(config-role)# interface ?
policy Configure the interface policy for this role
SW1(config-role)# interface policy ?
461

deny
Deny access to a interface unless specifically permitted
SW1(config-role)# interface policy deny

SW1(config-role-interface)# ?
no
permit Permit access to interfaces (applicable if interface policy is 'deny')
end
Go to exec mode
exit
pop
push
where
SW1(config-role-interface)# permit ?
interface Enter the range of interfaces accessible the role
SW1(config-role-interface)# permit interface ethernet3/1-10
Just like with normal interfaces you are able to configure interface ranges under the
configuration. Which is applied accordingly.
SW1(config-role-interface)# ?
no
permit Permit access to interfaces (applicable if interface policy is 'deny')
end
Go to exec mode
exit
pop
push
where
SW1(config-role-interface)# exit
SW1(config-role)# exit
Now finally after configuring the features, the role and the interface limitation we can
configure the user that is going to use this role.
SW1
SW1(config)# username user1 role USER1_ROLE password User1p@ssw0rd
SW1(config)#
Now we can verify the user and role configuration.

SW1(config)# show user-account user1
user:user1
this user account has no expiry date
roles:USER1_ROLE
SW1(config)# show role name USER1_ROLE
Role: USER1_ROLE
Description: new role
vsan policy: permit (default)
Vlan policy: permit (default)
Interface policy: deny
Permitted interfaces:
Ethernet3/1-10
Vrf policy: permit (default)
------------------------------------------------------------------Rule
Perm
Type
Scope
Entity
------------------------------------------------------------------1
permit read-write feature-group
USER1_FEATURES
462
SW1(config)# show role feature-group name USER1_FEATURES

feature group: USER1_FEATURES
vlan
svi
spanning-tree
hsrp
SW1(config)#
The best test is of course to log-in to the switch with this new username and verify what you
are able to configure now.
TerminalServer#ssh -l user1 172.16.100.2
Nexus 5000 Switch
Password: User1p@ssw0rd
SW1#
SW1#
SW1# ?
clear
Reset functions
configure Enter configuration mode
debug
Debugging functions
show
Show running system information
end
Go to exec mode
exit
SW1# show ?
spanning-tree
vlan
Show spanning tree information

Vlan commands
SW1# conf t
SW1(config)# ?
spanning-tree Spanning Tree Subsystem
vlan
Vlan commands
end
Go to exec mode
exit
SW1(config)#
463
As is shown in the above outputs. The options for viewing and configuring items is very limited.
This limitation is enforced by the role configuration so our first questions are completed.
Next up is the configuration of another user with other limitations.
This user should again be limited to a number of features. Plus its only allowed to review
settings and execute show commands.
We are not allowed to create another feature-group or configure the features for all the
routing protocols. This means that we should find another creative solution.
When we verify the feature-group configuration. We see that there is a default featuregroup available on the switch in the OS.
SW1(config)# show role feature-group
ERROR: Feature "router-isis" does not exist
feature group: L3
router-bgp
router-eigrp
router-ospf
router-rip
feature group: USER1_FEATURES
vlan
svi
spanning-tree
hsrp
SW1(config)#
As we need to configure the routing protocols we can use this default feature group for the
role configuration.
We
configure
the
role
for
this
next
user.
SW1
SW1(config)# role name USER2_ROLE
SW1(config-role)# rule 1 permit read
read
read-write
SW1(config-role)# rule 1 permit read feature-group ?
WORD Enter the access entity name (Max Size 32)
SW1(config-role)# rule 1 permit read feature-group L3
SW1(config-role)# rule 2 permit read feature access-list
SW1(config-role)# rule 3 permit read feature license
Just like with permitting only selected interfaces, we can also limit the configuration of certain
VRFs or VSANs. In this case we should limit the VRFs that the user is allowed to configure.
SW1
SW1(config-role)# vrf ?
policy Configure the vrf policy for this role
SW1(config-role)# vrf pol deny
464

SW1(config-role-vrf)# permit vrf
SW1(config-role-vrf)# exit
SW1(config)# username user2 role
SW1(config)#
VPN1
VPN2
VPN3
USER2_ROLE password User2User2
After configuring the username with the correct role assignment we finished the second
users role configuration.
The next user is only allowed to configure management protocols. This means that we should
create a feature-group consisting of all management protocols that we can find in the
overview and add those to the group together with the possibility to upgrade software of the
device.
aaa
arp
cdp
l3vm
ping
snmp
radius
syslog
tacacs
install
license
callhome
platform
access-list
svi
hsrp
igmp
msdp
vlan
dot1x
ipfib
eth-span
router-bgp
router-rip
ethanalyzer
(Ethernet Analyzer)
router-ospf
router-eigrp
spanning-tree
acl
sfm
sme
fcns
fcsp
fdmi
fspf
rlir
rscn
span
vsan
wwnm
zone
fcanalyzer
sme-kmc-admin
465

sme-stg-admin
The marked lines show all the management protocols that we can allow access to.
SW1
SW1(config)# role feature-group name
SW1(config-role-featuregrp)# feature
MGMT_PROTOCOLS
aaa
radius
tacacs
cdp
snmp
syslog
callhome
Next up we configure the role and we add the final step, which is to allow software upgrades.
We couldve also configured this together with the feature-group, but here we demonstrate
that we can both use a feature-group together with normal features in all types of
configuration.
SW1
SW1(config-role-featuregrp)# role name MAINTENANCE_ROLE
SW1(config-role)# rule 1 permit read-write feature-group MGMT_PROTOCOLS
SW1(config-role)# rule 2 permit read-write feature install
SW1(config)# username maintenance role MAINTENANCE_ROLE password MainTenanc3
SW1(config)#
Finally we create the username and we are done for this user. The installation of software
should of course be done with the install command, which is the only command the user has
access to.
The next user should only be allowed to configure Fibre Channel storage related features.
Again we check out the list of features and verify which should be added to a feature-group.
aaa
arp
cdp
l3vm
ping
snmp
radius
syslog
tacacs
install
license
callhome
platform
access-list
svi
hsrp
466

igmp
msdp
vlan
dot1x
ipfib
eth-span
router-bgp
router-rip
ethanalyzer
(Ethernet Analyzer)
router-ospf
router-eigrp
spanning-tree
acl
sfm
sme
fcns
fcsp
fdmi
fspf
rlir
rscn
span
vsan
wwnm
zone
fcanalyzer
sme-kmc-admin
sme-stg-admin
The marked lines show the storage related features that should be given access to. The final 3
features are related to a special storage feature, where dedicated users can only be given
access to these commands. They are not in scope of this configuration and are not necessary to
be configured.
SW1
SW1(config)# role feature-group name STORAGE
SW1(config-role-featuregrp)# feature sfm
SW1(config-role-featuregrp)# feature sme
SW1(config-role-featuregrp)# feature fcns
SW1(config-role-featuregrp)# feature fcsp
SW1(config-role-featuregrp)# feature fdmi
SW1(config-role-featuregrp)# feature fspf
SW1(config-role-featuregrp)# feature rlir
SW1(config-role-featuregrp)# feature rscn
SW1(config-role-featuregrp)# feature vsan
SW1(config-role-featuregrp)# feature wwnm
SW1(config-role-featuregrp)# feature zone
SW1(config-role-featuregrp)# feature fcanalyzer
SW1(config-role-featuregrp)# exit
SW1(config)#
SW1(config)#
SW1(config)# role name STORAGE_ROLE
SW1(config-role)# rule 1 permit read-write feature-group STORAGE
467
After configuring the role and the feature-group. We complete the configuration by creating
the username attached to the role.
SW1
SW1(config)# username storage-admin role STORAGE_ROLE password st0rage-@Dmin
SW1(config)#
The final role that should be configured is a NOC user. This user can have all rights like the one
of the network-operator, but is also allowed to issue Telnet and SSH commands from
the box.
Here we need to use command authorization and wildcards!
SW1
SW1(config)# role name NOC
SW1(config-role)# rule 1 permit command show
SW1(config-role)# rule 2 permit command telnet *
SW1(config-role)# rule 3 permit command ssh *
SW1(config)# username nocuser password NOCus3r role NOC
SW1(config)#
Now we verify if the user indeed has the correct rights.

TerminalServer#ssh -l nocuser 172.16.100.2
Nexus 5000 Switch
Password: NOCus3r
SW1# conf t
We are allowed to ensure configuration mode, but we cannot change anything in here.
SW1(config)# ?
no
username Configure user information.
end
Go to exec mode
exit
SW1(config)# exit
SW1# show ?
aaa
access-lists
accounting
banner
Show
List
Show
Show
aaa information
access lists
accounting configuration
current motd banner message
468

boot
callhome
cdp
cfs
checkpoint
cimserver
class-map
cli
clock
Show Bootvar Variables

Show callhome information
Show Cisco Discovery Protocol information
CFS Show Command handler
Show configuration rollback checkpoints
Show cimserver configuration
Show class maps
Show CLI information
Display current Date
All show commands are available!

SW1# telnet ?
A.B.C.D Enter a valid IPv4 address
WORD
Enter hostname (Max Size 64)
SW1# telnet
SW1# ssh ?
WORD Enter hostname or user@hostname (Max Size 64)
SW1# ssh
And the user is able to access Telnet and SSH features.

The final question of this task is to ensure that all switches share the same role configuration.
The big advantage here is that RBAC supports distribution through CFS and therefore the
configuration is easily done by enabling CFS distribution and ensuring all role configuration is
available across all switches.
Pay attention as the distribute command is not automatically completed by the CLI, but it does
exist and is supported.
SW1
SW1(config)# role distribute
SW2
SW3
The final step is to distribute all roles of SW1 to the switches by issuing a commit.
SW1
SW1(config)# role commit
2012 Nov 20 21:18:55 SW1 %CFS-2-MTS_REJECT: Verification failed reject MTS message
SAP 37:RR-token 0x3fe696b
Please check Status with show command
SW1(config)#
SW1(config)# show role status
469

Distribution: Enabled
Session State: Locking
SW1(config)#
Now we finished the first task of this management chapter!
Task 2: Traffic monitoring
This task is about configuring monitoring sessions or SPAN sessions. Monitoring sessions are
configurations for monitoring traffic on certain interfaces or VLANs.
This is used for troubleshooting purposes and for interception purposes. Traffic is fully copied
from one interface to another.
These sessions are called SPAN sessions. You can monitor traffic from the following sources:
Ethernet interfaces
Port Channel interfaces
Control-Plane interface to the forwarding plane
VLANs
Satellite and port-channel interfaces on Fabric Extenders
The traffic is then copied to a destination interface. Multiple sources can be sent to the same
destination interface. Be aware that its possible to oversubscribe the destination interface with
configuring multiple sources.
There are ways to prevent that destination interfaces are oversubscribed, by configuring rate
limiting or truncation of packets.
All Nexus switches support up to 16 SPAN sessions to be configured. Only 2 can be active
simultaneous.
Virtual SPAN sessions are sessions that do not monitor a full interface, but only specific
VLANs on this physical interface.
Our configuration starts with configuring these Virtual SPAN sessions on the port-channels
connecting
to
SW2
and
SW3.
Lets start with configuring the switch.
SW1
SW1(config)# monitor session 1
SW1(config-monitor)# source interface port-channel101
SW1(config-monitor)# source interface port-channel102
470
First configure the sources which are the port-channels.

SW1(config-monitor)# filter vlan 50,99
Then we filter out the VLANs that we really need.

SW1(config-monitor)# destination interface ethernet3/19
The destination interface is specified. This can not be a port-channel member, only a physical
interface
SW1(config-monitor)# no shut
By default are SPAN sessions in a shutdown state, so they need to be enabled.

SW1(config-monitor)# int e3/19
SW1(config-if)# sw monitor
SW1(config-if)#
Finally we should configure the destination interface to support the SPAN traffic, which it by
default wont.
We verify the working of the session and if we configured everything properly.
The interface will remain down as there is no cable connected, so the session will also show up
as being down, which is not a problem for our task as it is the expected result.
SW1(config-if)# do sh int e3/19
Ethernet3/19 is down (SFP not inserted)
Hardware: 10000 Ethernet, address: 0005.9b73.5752 (bia 0005.9b73.5752)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
Encapsulation ARPA
Port mode is trunk
auto-duplex, 10 Gb/s
Input flow-control is off, output flow-control is off
Switchport monitor is on
SW1(config-if)# show monitor session 1
session 1
--------------type
: local
state
: down
source intf
:
rx
: Po101,Po102
tx
: Po101,Po102
both
: Po101,Po102
source VLANs
:
rx
: 50,99
source VSANs
:
rx
:
destination ports : Eth3/19
471

Legend: f = forwarding enabled, l = learning enabled
SW1(config-if)#
The next question is related to changing the MTU size of the monitoring traffic. This is done to
prevent the monitoring server for having too much traffic sent at too large MTU sizes.
The packet after the MTU size which is configured is deleted, so not the entire packet is sent to
the destination interface.
SW1
SW1(config-monitor)# mtu 1100
Now the MTU size will always be the same and never larger, not dependent on the source
packet.
The next question asks for another approach. The question asks for transportation of the traffic
across the network.
This technology is called Remote SPAN or RSPAN. Its been around for a long time and it means
that traffic being monitored on one switch, is sent into a special dedicated VLAN which is then
picked up at another switch where the destination interface is located.
This is exactly what we need for this next question.
The traffic is being monitored on another switch and is sent to SW1 where the destination
interface is located.
SW1
SW1(config-vlan)# remote-span
This is the trick for RSPAN sessions. The VLAN should be marked with the remote-span
keyword. Then the traffic is correctly picked up and received well.
SW1(config-monitor)# source vlan 601
SW1(config-monitor)# destination interface ethernet3/20
The configuration of the RSPAN session is equal to a normal session for the rest. Keep in mind
to use another number for the session so you do not overlap with the previous question.

SW1(config-if)#
472
Now the traffic is being picked up and sent to the destination interface.
The next question is related to a technology introduced in the Nexus platform. Instead of
transporting traffic across a Layer 2 domain its now also possible to transport it across a
layer 3 domain inside a GRE tunnel.
This technology is called ERSPAN. It supports transport across a routed Layer 3 network.
The Nexus 5000 switches only support Source based sessions where as the Nexus 7000 also
supports destination sessions.
There is no specification on which IP addresses should be used for this session. So we will use
any routed network between the switches. For example VLAN 99 which is configured in a
previous chapter.
Then the configuration is pretty straight forward by just configuring the right IP addresses,
interfaces and some special values related to the ERSPAN session.
SW2
SW2(config)# monitor session 3 type erspan-source
The type of session is very important. Otherwise the ERSPAN configuration parameters could
not be configured.
SW2(config-monitor)# source interface ethernet1/17
SW2(config-monitor)# destination ip 198.18.99.1
SW2(config-monitor)# erspan-id 100
The ERSPAN ID is a unique identifier to ensure the ERSPAN session is not overlapped with
another session, which could cause mistakes in traffic monitoring.
SW2(config-monitor)# vrf default
We should also configure the source address that the switch uses, as that is required for
configuration on the destination switch.
SW2(config)# monitor erspan origin ip-address 198.18.99.2
SW2(config-erspan-src)# exit
Next up is the configuration of the destination.

SW1
SW1(config)# monitor
SW1(config-monitor)#
session 3 type erspan-destination

source ip 198.18.99.2
destination interface ethernet3/17
erspan-id 100
The ERSPAN ID should match on both sides of the SPAN session.

SW1(config-monitor)# vrf default
473

SW1(config-if)#
Then the destination interface should be marked as a SPAN destination port and were done!
The final tasks are related to tuning the ERSPAN configuration.
The first is about giving priority to ERSPAN traffic. This is done by giving the ERSPAN traffic a
higher DSCP bit, making it a higher priority traffic in Quality of Service configurations.
This configuration is done on the source.
SW2
SW2(config)# monitor session 3 type erspan-source
SW2(config-monitor)# ip dscp 11
Finally the granularity should be configured. This means in how much precision the traffic is
monitored. Configuration changes for this are done on the source as well.
SW2
SW2(config)# monitor erspan granularity ns
Nanosecond granularity is the finest we can get.

With this configuration we finished the task about SPAN ports!
Task 3: NetFlow
In this task about NetFlow we will be configuring traffic monitoring based on the flow of
packets. This information is then used to form statistics about the network usage.
NetFlow works in a way that after its been enabled it will look for various packet streams that
have identical packet headers. Meaning they belong to a flow. This flow is a unidirectional
stream of packets. The flows are temporarily saved in the NetFlow cache en is then
exported to a Flow server where it can be used for analysis and accounting on network
traffic.
This process can take up a lot of resources. Which is why its done in hardware on the Nexus
platforms. NetFlow uses a sampling rate to capture traffic according to a configured rate. For
example every 1000 packets one is picked out and analyzed. The higher the sampling rate is,
the more precise the NetFlow information becomes, but also the required resources on the
device
are
getting
higher.
474
There are several versions of the NetFlow export protocol. The Nexus switches support
version 5 and version 9. Where the highest version is recommended due to flexibility and
support for IPv6 and layer 2 information.
NetFlow support is very specific to the platform and the hardware. Currently only the Nexus
7000 supports the NetFlow feature.
The feature works by configuring various profiles. These profiles are bundled together
and applied to an interface. On this interface the specified information is then monitored.
We start by configuring the Nexus 7000 switch for the NetFlow record that the question
states.
The initial task is about creating a flow record based on IP address information. Together
with that the Flow ID and packets per second should be captured.
SW1
SW1(config)# feature netflow
SW1(config)# flow record IPX
SW1(config-flow-record)# ?
collect
Specify a non-key field
description Provide a description for this Flow Record
match
Specify a key field
no
end
Go to exec mode
exit
pop
push
where
SW1(config-flow-record)# match ?
datalink
Datalink (Layer 2) attributes
ip
IP attributes
ipv4
IPv4 attributes
ipv6
IPv6 attributes
transport Transport layer fields
SW1(config-flow-record)# match ipv4 ?
destination Destination Address
source
Source Address
SW1(config-flow-record)# match ipv4 source ?
address Address
SW1(config-flow-record)# match ipv4 source address
SW1(config-flow-record)# match ipv4 destination address
We first configure by which fields this record should be matched. Which are the IPv4
addresses like our question stated.
SW1
SW1(config-flow-record)# collect flow ?
sampler Sampler
475
SW1(config-flow-record)# collect flow samp ?

id Identifer for sampler used for the flow
SW1(config-flow-record)# collect flow samp id
Then the NetFlow ID should be captured.

SW1
SW1(config-flow-record)# collect ?
counter
Counters to collect
flow
Flow identifying fields
ip
IP attributes
routing
Routing attributes
timestamp Timestamp fields
transport Transport layer fields
SW1(config-flow-record)# collect counter ?
bytes
Total number of bytes
packets Total number of packets
SW1(config-flow-record)# collect counter packets ?
<CR>
long
Long counter (64 bits)
SW1(config-flow-record)# collect counter packets long
As well as the Packets per Second counter. Be aware that you need to use 64-bit counters
as the question stated.
SW1(config-flow-record)#
In the next question it is asked that the information should be exported to a dedicated flow
server. Read ahead on the next question, because you will need some crucial information to
configure this export.
The last question of this task states that we should be able to export Layer 2 fields to the
Flow Server.
This automatically means that we need to use NetFlow version 9 in the Exporter
configuration.
SW1
SW1(config)# flow exporter IPX_EXPORT
SW1(config-flow-exporter)# version ?
5 Version 5 Export
9 Version 9 Export
SW1(config-flow-exporter)# version 9
SW1(config-flow-exporter-version-9)# ?
no
option
Version 9 Option Templates and Data
template Version 9 Template
end
Go to exec mode
exit
pop
push
476

where
SW1(config-flow-exporter-version-9)# exit
SW1(config-flow-exporter)# ?
description Provide a description for this Flow Exporter
destination Specify the destination address
dscp
Optional DSCP
no
source
Source Interface for this destination
transport
Transport Destination Port
version
Specify the export version
end
Go to exec mode
exit
pop
push
where
SW1(config-flow-exporter)# destination 172.16.100.109 ?
<CR>
use-vrf Optional VRF label
SW1(config-flow-exporter)# destination 172.16.100.109
SW1(config-flow-exporter)# exit
SW1(config)#
After configuring the version and the destination we finished configuring our exporter.
Now the configuration hasnt finished!
We continue with configuring the sampling rate of the packets.
SW1
SW1(config)# flow ?
exporter Define a
monitor
Define a
record
Define a
timeout
Define a
Flow
Flow
Flow
Flow
Exporter
Monitor
Record
Timeout
SW1(config)# sampler IPX_SAMPLE

SW1(config-flow-sampler)# ?
description Provide a description for this Flow Sampler
mode
Define the Sampler mode
no
end
Go to exec mode
exit
pop
push
where
SW1(config-flow-sampler)# mode ?
<1-64> Number of samples per sampling - 64 max
SW1(config-flow-sampler)# mode 5 ?
out-of M out of N packets
SW1(config-flow-sampler)# mode 5 out-of ?
<1-8192> Number of packets in each sampling - 8192 max
SW1(config-flow-sampler)# mode 5 out-of 150
SW1(config-flow-sampler)# exit
SW1(config)#
477
Now we configured our sampling profile for 5 out of 150 packets to be sampled.
Finally we should combine all this information together so the port-channels, like stated in the
first
question,
are
monitoring
flows
and
exporting
them
correctly.
SW1
SW1(config)# flow monitor IPX_MON
SW1(config-flow-monitor)# ?
description Provide a description for this Flow Monitor
exporter
Add an Exporter to use to export records
no
record
Specify Flow Record to use
end
Go to exec mode
exit
pop
push
where
SW1(config-flow-monitor)# exporter IPX_EXPORT
SW1(config-flow-monitor)# record IPX
SW1(config-flow-monitor)# exit
SW1(config)#
Within a monitor configuration, all items are combined together (exporter and record).
SW1
SW1(config)#
SW1(config)# int port-chan101
SW1(config-if)# ip flow monitor ?
WORD Name of Flow Monitor (Max Size 63)
SW1(config-if)# ip flow ?
monitor Apply a Flow Monitor to this interface
SW1(config-if)# ip flow monitor IPX_MON ?
input
Apply Flow Monitor on input traffic
output Apply Flow Monitor on output traffic
SW1(config-if)# ip flow monitor IPX_MON input ?
<CR>
sampler Optional Sampler to apply to this Flow Monitor
Finally the flow monitor should be configured under the interface where its necessary. Do
not forget to apply the sampler which we just configured!
SW1
SW1(config-if)# ip flow monitor IPX_MON input sampler ?
WORD Name of Sampler (Max Size 63)
SW1(config-if)# ip flow monitor IPX_MON input sampler IPX_SAMPLE
478
There was no direction specified in which the flow monitor should be applied so we are going
to configure it on both directions.
SW1
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
SW1(config-if)#
ip flow monitor IPX_MON output sampler IPX_SAMPLE

int port-chan102
ip flow monitor IPX_MON input sampler IPX_SAMPLE
ip flow monitor IPX_MON output sampler IPX_SAMPLE
The final step is to apply the flow configuration to both directions and to both interfaces.
And after the application of the interfaces. We finished the configuration of NetFlow!
Of course there are many other combinations that can be configured, but for now we have
configured all properties, which should be enough for you to understand all variants that are
possible within the configuration.
Task 4: Management protocols
This next task will be about configuring all the various management protocols that are available
on the Nexus platform. The management protocols that we are configuring are the following.
SNMP
Syslog
NTP
CDP
These are no surprises as these are the management protocols which have been around for a
long time and are configured on many different platforms and software versions.
There is one important difference between IOS and NX-OS and that is that SNMP version 3 is
by default enabled on the device. This is done, because the Data Center Network
Manager (DCNM) uses SNMP v3 to communicate with the Nexus switches.
When you create a user-account the device automatically creates a SNMP version 3 user
account, making it possible to log-in using SNMPv3 as well.
Still NX-OS is compatible with SNMPv2c, because the entire world is still using it to manage
their networks.
We start by configuring the classical SNMP environment for our switch.
SW1
SW1(config)# snmp-server host ?
A.B.C.D|A:B::C:D|WORD IPv4 or IPv6 address or DNS Name of SNMP notification host
479

SW1(config)# snmp-server host 172.16.100.110 ?
WORD
SNMP community string or SNMPv3 user name (Max Size 32)
filter-vrf
Filters notifications to the notification host receiver based
on the configured
VRF
informs
Send Inform messages to this host
source-interface Source interface to be used for sending out SNMP notifications
to this host
traps
Send Traps messages to this host
use-vrf
Configures SNMP to use the selected VRF to communicate with the
host receiver
version
SNMP version to use for notification messages
SW1(config)# snmp-server host 172.16.100.110 traps ?
WORD
SNMP community string or SNMPv3 user name (Max Size 32)
version SNMP version to use for notification messages
Ensure traps are sent to the host as the question stated.

SW1(config)# snmp-server host 172.16.100.110 traps version ?
1
Use SNMPv1
2c Use SNMPv2c
3
Use SNMPv3
SW1(config)# snmp-server host 172.16.100.110 traps version 2c ?
WORD SNMP community string or SNMPv3 user name (Max Size 32)
We also need to configure the version at which the traps should be sent. In this case we will be
using the Version 2c.
SW1
SW1(config)# snmp-server host 172.16.100.110 traps version 2c IPexpert
SW1(config)# snmp-server enable traps ?
<CR>
aaa
Module notifications enable
bridge
Enable SNMP STP Bridge MIB traps
callhome
cfs
config
entity
fcdomain
fcns
fcs
fctrace
fdmi
feature-control Module notifications enable
fspf
license
link
rf
rmon
rscn
scsi
snmp
stpx
Enable SNMP STPX MIB traps
sysmgr
upgrade
vsan
vtp
zone
480

SW1(config)# snmp-server enable traps
SW1(config)#
Before the switch will actually send traps. They need to be enabled first. By default the switch
will not send any traps, so then the question would not be answered correctly.
Ensure that traps are enabled when is stated in the question that a server needs to receive
traps. Its not only the possibility, but it actually needs to receive them.
In
this
case
we
enable
all
the
possible
traps
The next question specifies that the server should also be able to access the switch via a specific
community string.
SW1
SW1(config)# snmp-server community ?
WORD SNMP community string (Max Size 32)
SW1(config)# snmp-server community IPexpert ?
<CR>
group
Group to which the community belongs
ro
Read-only access with this community string
rw
Read-write access with this community string
use-acl Acl name to filter snmp requests
SW1(config)# snmp-server community IPexpert ro
Read the question carefully as it clearly specified that the server should be able to read
information. So we can configure the read-only feature here.
The server is now able to access the switch to read out all information and it will receive traps.
The next questions are related to some contact information which is used in management
systems to identify devices based on their location. Its a free text format, so you can fill out
anything you like.
SW1
SW1(config)# snmp-server location Utrecht, Netherlands, Europe
SW1(config)# snmp-server ?
aaa-user
Set duration for which aaa-cached snmp user exists
community
Set community string and access privs
contact
Modify sysContact
context
SNMP context to be mapped
enable
Enable SNMP Traps
globalEnforcePriv Globally enforce privacy for all the users
host
Specify hosts to receive SNMP notifications
location
Modify sysLocation
mib
Mib access parameters
protocol
Snmp protocol operations
source-interface
Source interface to be used for sending out SNMP notifications
tcp-session
Enable one time authentication for snmp over tcp session.
user
Define a user who can access the SNMP engine
481

SW1(config)# snmp-server con
contact
context
SW1(config)# snmp-server contact Rick Mur, IPexpert
SW1(config)#
As the next question states, we want the switch to enforce encryption for SNMPv3 traffic. One
of the biggest complaints about SNMPv1 and SNMPv2c is that it has no support for any
authentication other than the community strings and that it has no support for encryption.
SNMPv3 offers all of that, but its not required by default.
With a single command we can turn the switch so it will only accept encrypted SNMPv3
requests.
SW1
SW1(config)# snmp-server ?
aaa-user
Set duration for which aaa-cached snmp user exists
community
Set community string and access privs
contact
Modify sysContact
context
SNMP context to be mapped
enable
Enable SNMP Traps
globalEnforcePriv Globally enforce privacy for all the users
host
Specify hosts to receive SNMP notifications
location
Modify sysLocation
mib
Mib access parameters
protocol
Snmp protocol operations
source-interface
Source interface to be used for sending out SNMP notifications
tcp-session
Enable one time authentication for snmp over tcp session.
user
Define a user who can access the SNMP engine
SW1(config)# snmp-server globalEnforcePriv
SW1(config)#
We then should configure a user for SNMPv3 with rights. Now here is an overlap with the RBAC
configuration as SNMPv3 uses roles from RBAC to give access to what the management device
can access.
When you create a username, the switch will automatically create an SNMPv3 username with
the same rights as well. You can individually delete the SNMPv3 usernames when you like, but
its not possible to turn it off that the user is automatically created in the first place.
When an SNMPv3 user is created it also works vice versa. It will automatically create a useraccount as well.
SW1
SW1(config)# snmp-server user ?
WORD Name of the user (Max Size 32)
SW1(config)# snmp-server user version3 ?
<CR>
WORD
Group name (ignored for notif target user) (Max Size 32)
auth
Authentication parameters for the user
SW1(config)# snmp-server user version3 auth ?
md5 Use HMAC MD5 algorithm for authentication
482

sha
Use HMAC SHA algorithm for authentication
SW1(config)# snmp-server user version3 auth md5 ?

WORD Authentication password for user (Max Size 130)
First we need to specify the password of the user.

SW1(config)# snmp-server user version3 auth md5 version3password ?
<CR>
engineID
EngineID for configuring notif target user (for V3 informs)
localizedkey Specifies whether the passwords are in localized key format
priv
Encryption parameters for the user
SW1(config)# snmp-server user version3 auth md5 version3password priv ?
WORD
Privacy password for user (Max Size 130)
aes-128 Use 128-bit AES algorithm for privacy
SW1(config)# snmp-server user version3 auth md5 version3password priv
version3password ?
<CR>
engineID
EngineID for configuring notif target user (for V3 informs)
localizedkey Specifies whether the passwords are in localized key format
The next step is to configure the encryption key that is used. There is no problem in using the
same password for both keys. Which is also the default when a normal username is created.
SW1
SW1(config)# snmp-server user version3
version3password
SW1(config)# snmp-server user version3
<CR>
WORD
Group name (ignored for notif
auth
Authentication parameters for
auth md5 version3password priv

?
target user) (Max Size 32)
the user
SW1(config)# snmp-server user version3 storage-admin
Finally the user should be given the same rights as a currently existing user role. Now at first it
seems strange where to configure this, but the roles are called groups in this configuration.
You should use normal role names as the group name in this user configuration. After which
it will take over the rights the user has and work as the question stated.
Now we finished the part for configuring SNMP related configurations.
We can verify if everything is configured at is should.
SW1
SW1(config)# show snmp ?
<CR>
>
>>
community
Show snmp community strings
context
Show snmp context mapping entries
engineID
Show snmp engineID
group
Show snmp group
host
Show snmp hosts
483

internal
sessions
source-interface
trap
user
|
Show
Show
Show
Show
Show
Pipe
snmp internal information

snmp sessions
source-interface through which notifications are sent
snmp traps
SNMPv3 users
command output to filter
SW1(config)# show snmp user

______________________________________________________________
SNMP USERS [global privacy flag enabled]
______________________________________________________________
User
____
Auth
____
Priv(enforce) Groups
_____________ ______
rick
md5
des(no)
network-admin
admin
md5
des(no)
network-admin
nocuser
md5
des(no)
network-operator
NOC
readonly
md5
des(no)
network-operator
version3
md5
des(no)
network-operator
storage-admin
maintenance
md5
des(no)
MAINTENANCE_ROLE
storage-admin
md5
des(no)
STORAGE_ROLE
______________________________________________________________
NOTIFICATION TARGET USERS (configured for sending V3 Inform)
______________________________________________________________
User
____
Auth
____
Priv
____
Here its shown that all the users that have a regular user account are also configured as
SNMPv3 users.
SW1
SW1(config)# show snmp host
------------------------------------------------------------------Host
Port Version Level Type
SecName
------------------------------------------------------------------172.16.100.110
162 v2c
noauth trap
IPexpert
Here you can verify which servers receive the traps that the switch sends.
SW1(config)# show snmp community
Community
Group / Access
---------------------IPexpert
network-operator
SW1(config)#
context
-------
acl_filter
----------
484
Finally you can verify which SNMP communities are configured on the switch and how they are
operating.
The next management protocol which we will be configuring is the Syslog protocol. This
protocol is used for troubleshooting and debug messages and not so much for alarming.
For alarming SNMP traps are used and Syslog is used mostly during troubleshooting processes.
There are several levels of message priority you are able to see and which you are able to
process in separate log files when desired.
There are 7 levels of syslog priorities.
0: Emergency
1: Alert
2: Critical
3: Error
4: Warning
5: Notification
6: Informational
7: Debugging
All messages that the system can give have a priority attached. Usually up to level 4 is
shown as errors of the system. Level 5 and level 6 are messages related to processes
that perform actions. For example an OSPF neighbor comes online. When a neighbor goes
down, its possibly an error so this gets a higher priority.
Level 7 is only used for debug messages and is therefore never shown. In IOS they were
always shown on the Console port, but this could cause serious issues as every console
character had impact on the CPU. Therefore in NX-OS this has been disabled and you need to
turn logging for debugging specifically on.
By default the following levels of logging are enabled. The logging monitor is the logging which
is shown in Telnet and SSH sessions.
SW1
SW1(config)# show logging
Logging
Logging
Logging
Logging
Logging
Logging
Logging
console:
monitor:
linecard:
fex:
timestamp:
server:
logfile:
enabled (Severity: critical)

enabled (Severity: notifications)
Seconds
disabled
enabled
485
Currently its not even supported to raise the logging level of the console, without having
changed the console speed itself.
SW1(config)# logging console 6
Baud rate of console should be at least 38400 to increase logging level
The first Syslog question is so that Telnet and SSH sessions see informational level
messages. This is done by changing the logging level of monitor sessions.
SW1
SW1(config)# logging monitor ?
<CR>
<0-7> 0-emerg;1-alert;2-crit;3-err;4-warn;5-notif;6-inform;7-debug
SW1(config)# logging monitor 6
With the question mark its verifiable which level should be configured on the logging level.
Logging
Logging
Logging
Logging
Logging
Logging
Logging
console:
monitor:
linecard:
fex:
timestamp:
server:
logfile:

enabled (Severity: information)
Seconds
disabled
enabled
Now the informational messages are logged in Telnet and SSH sessions.
The next question states that debugging messages should be logged into a logfile.
SW1
SW1(config)# logging ?
abort
Flushes cached data without committing and releases the lock
commit
Commits cached data (of all msg types) and releases the lock
console
Set console logging
distribute Enables/disables fabric distribution using cfs.
event
Interface events
level
Facility parameter for syslog messages
logfile
Set File logging
message
Interface events
module
Set module logging
monitor
Set terminal line(monitor) logging level
server
Enable forwarding to Remote Syslog Server
timestamp
Set logging timestamp granularity
SW1(config)# logging logfile ?
WORD Enter the logfile name
SW1(config)# logging logfile DEBUG ?
SW1(config)# logging logfile DEBUG 7
SW1(config)#
486
We can configure logfiles in which we can enable separate severities to be logged.

Logging
Logging
Logging
Logging
Logging
Logging
Logging
console:
monitor:
enabled (Severity: information)
linecard:
fex:
timestamp:
Seconds
server:
disabled
logfile:
enabled
Name - DEBUG: Severity - debugging Size - 4194304
Next we should be ensuring that the log files are marked with timestamps that are very
precise.
SW1
SW1(config)# logging timestamp ?
microseconds Timestamp in micro-seconds
milliseconds Timestamp in milli-seconds
seconds
Timestamp in seconds (Default)
SW1(config)# logging timestamp microseconds
SW1(config)#
Microseconds is the most precise value we can use for our log messages. From now on the log
messages are logged with timestamps including a second, millisecond and microsecond value.
Finally we should configure a Syslog server which should receive all notifications that are
generated by the switch.
SW1
SW1(config)# logging ?
abort
Flushes cached data without committing and releases the lock
commit
Commits cached data (of all msg types) and releases the lock
console
Set console logging
distribute Enables/disables fabric distribution using cfs.
event
Interface events
level
Facility parameter for syslog messages
logfile
Set File logging
message
Interface events
module
Set module logging
monitor
Set terminal line(monitor) logging level
server
Enable forwarding to Remote Syslog Server
timestamp
Set logging timestamp granularity
SW1(config)# logging server 172.16.100.110 ?
<CR>
We should specify the correct level, which is 5 in this case where we want the
notifications to be forwarded to the server.
SW1(config)# logging server 172.16.100.110 5 ?
<CR>
facility Facility to use when forwarding to server
487

use-vrf
Enter VRF name, default is default VRF
SW1(config)# logging server 172.16.100.110 5 facility ?

auth
Use auth facility
authpriv Use authpriv facility
cron
Use Cron/at facility
daemon
Use daemon facility
ftp
Use file transfer system facility
kernel
Use kernel facility
local0
Use local0 facility
local1
Use local1 facility
local2
Use local2 facility
local3
Use local3 facility
local4
Use local4 facility
local5
Use local5 facility
local6
Use local6 facility
local7
Use local7 facility
lpr
Use lpr facility
mail
Use mail facility
news
Use USENET news facility
syslog
Use syslog facility
user
Use user facility
uucp
Use Unix-to-Unix copy system facility
The last step is configuring the correct facility that should be used. Facilities in Syslog
are used for grouping messages together on the server itself and have no technical feature.
SW1
SW1(config)# logging server 172.16.100.110 5 facility local3 ?
<CR>
facility Facility to use when forwarding to server
use-vrf
Enter VRF name, default is default VRF
SW1(config)# logging server 172.16.100.110 5 facility local3
SW1(config)#
488
This finalizes the syslog configuration.

Next up is the configuration of the Network Time Protocol (NTP). This protocol is widely
used for distributing time information throughout a network. Many things rely on the
configuration of the correct timing, its especially necessary to have synchronized timing on
your network devices, because you require correct timestamps when you are troubleshooting a
problem.
The NTP protocol is separated in servers, clients, peers and masters. The servers are
authoritative over their connecting clients, but do not hold a master clock. That is only for the
clocks configured as master. Peers are devices which are synchronizing towards each other
and neither of them is really authoritative.
Not all switches can become master clocks. In the case of the Nexus family. Only the Nexus
7000 can become a master clock.
The trust worthiness of a clock is dependent on its stratum. This can be seen as a hopcount as each time a server is in between, the stratum count increases of the clock. This
means that the lower the number, the closer the clock is to the source.
The first 2 questions are creating a master, server and 2 clients. It means that SW1 should be
the master clock for the network and SW2 and SW3 are the clients that synchronize their time
against it.
SW1
SW1(config)# ntp ?
abort
authenticate
authentication-key
commit
distribute
enable
logging
peer
server
source
source-interface
trusted-key
Abort the ntp configuration

Enable/Disable authentication
NTP authentication key
Commit the ntp configuration
Enable NTP configuration distribution
Enable NTP
Enable/Disable logging of NTPD Events
NTP Peer address
NTP server address
Source of NTP packets
Source interface sending NTP packets
NTP trusted-key
SW1(config)# ntp master 1
The master clock is now configured, so we can continue with the clients. We can use any
Layer 3 address active on the switch for this use. So we will be using the VLAN 99 IP
addresses.
SW2
SW3
489
After the configuration we can verify that the time is synchronizing against the master clock.
Keep in mind that this can take a long time sometimes. During the CCIE Lab its even better to
move on to the next task as you might need to wait up to 10 minutes before the NTP has been
synchronized correctly.
SW2(config)# show ntp peer-status
Total peers : 1
* - selected for sync, + - peer mode(active),
- - peer mode(passive), = - polled in client mode
remote
local
st
poll
reach delay
-----------------------------------------------------------------------=198.18.99.1
0.0.0.0
16
16
0
0.00000
SW1(config)#
We should now protect SW1 against unwanted NTP clients to be synchronizing time with us.
SW1
SW1(config)# ip access-list NTP_CLIENTS
SW1(config-acl)# permit ?
<0-255> A protocol number
ahp
Authentication header protocol
eigrp
Cisco's EIGRP routing protocol
esp
Encapsulation security payload
gre
Cisco's GRE tunneling
icmp
Internet Control Message Protocol
igmp
Internet Group Management Protocol
ip
Any IP protocol
nos
KA9Q NOS compatible IP over IP tunneling
ospf
OSPF routing protocol
pcp
Payload compression protocol
pim
Protocol independent multicast
tcp
Transmission Control Protocol
udp
User Datagram Protocol
SW1(config-acl)#
SW1(config-acl)#
SW1(config-acl)#
SW1(config)# ntp
permit ip host 198.18.99.2 any

permit ip host 198.18.99.3 any
exit
access-group serve NTP_CLIENTS
The serve keyword enables the NTP server to filter out requests from various connecting NTP
systems. When serve is used, it will only accept requests and will never synchronize its own
time against the specified IP addresses in the ACL.
To protect the NTP process even more, the requests should be authenticated!
SW1
SW1(config)# ntp authentication-key 1 md5 TimeIPX
SW1(config)# ntp trusted-key 1
SW1(config)# ntp authenticate
490
We first configure the master clock for authentication. Do not forget to enable the
authentication process itself, otherwise NTP will continue without authenticating anything.
The clients should be configured a little different, because they should authenticate against the
server.
SW2
SW2(config)# ntp server 198.18.99.1 key 1
SW3
SW3(config)# ntp server 198.18.99.1 key 1
Here we finished the configuration of NTP.

The last time related question is that we should set the timezone to our current location. This is
very important when you are using Time of Day time synchronization like NTP.
SW1
SW1(config)# clock timezone CET 1 0
SW1(config)#
SW2
SW2(config)#
SW3
SW3(config)#
You first configure a name which identifies the timezone that you are in. Then you specify the
time offset in hours and minutes that is different from GMT time.
491
The final questions in this management task are related to a protocol proprietary to Cisco. This
protocol is known as the Cisco Discovery Protocol.
It is used to exchange information about the Cisco devices which are connected to the
interfaces. Its extremely useful in your network environment as you immediately know what is
connected behind an interface and also at which interface on that other device. Additionally
you can even see what kind of device it is and when possible the management IP address to
connect to.
Its also extremely dangerous when used in environments where a lot of third parties connect,
like Service Provider environments or Multi-Tenant Data Centers.
We should first set SW1 to identify itself via its serial number compared to its hostname
which is the default.
SW1
SW1(config)#
advertise
enable
format
holdtime
timer
cdp ?
Highest CDP version supported on the switch
Enable/disable CDP on all interfaces
Device ID format for CDP
CDP hold time advertised (in seconds)
CDP refresh time interval (in seconds)
SW1(config)# cdp format?

format Device ID format for CDP
SW1(config)# cdp format ?
device-id Device ID format for CDP
SW1(config)# cdp
mac-address
serial-number
system-name
format device-id ?
Mac-address of the Chassis
Chassis Serial Number/OUI
System name/Fully Qualified Domain Name (Default)
SW1(config)# cdp format device-id serial-number

SW1(config)#
This is of course totally unusable in a production world, but in this case its what the question
asks for.
Next we should configure that switches should not use the default 60 second advertisements
of CDP packets, but should change this to 10 second intervals.
SW1
SW1(config)# cdp timer 10
SW1(config)#
SW2
SW2(config)#
492
SW3
SW3(config)#
And the last question is that we want to disable CDP to be enabled on certain interfaces that
are connected to a third party.
SW2
SW2(config-if-range)# no cdp ?
enable Enable/disable CDP on the interface
SW2(config-if-range)# no cdp enable
SW3
SW3(config-if-range)# no cdp enable
If you want to disable CDP to run on the entire device you should globally disable it with the
command no cdp run.
Herewith we finished the task about management protocols!
Task 5: Device management
The fifth task is not about management protocols, but about managing the device itself and
that means how file systems are handled, how configuration can be backed up (archived) and
other features related to management of the device and the command-line of NX-OS.
In the first task we are asked to store configuration of the switch so it can be re-used. Now this
can be done in multiple ways. One of them is by just storing the configuration on the flash.
This is not what we want, because we are also required to let it be compared with the current
(new) configuration.
NX-OS knows a special feature for this, called rollback. Its not the same as in other
operating systems like IOS-XR, where the configuration is edited in a text file and then applied
to the device.
In the NX-OS we can create configuration checkpoints. In which the configuration is saved.
Then you are able to compare the differences with a newer configuration and when necessary
even perform a rollback to the previous configuration.
SW1
SW1# checkpoint ?
<CR>
493

WORD
description
file
Checkpoint name (Max Size 32)

Checkpoint description for the given checkpoint
Create configuration rollback checkpoint to file
SW1# checkpoint CONFIG_BACKUP ?

<CR>
description Checkpoint description for the given checkpoint
SW1# checkpoint CONFIG_BACKUP
....Done
SW1#
After the checkpoint has been generated its possible to compare differences with the other
configuration files.
SW1
SW1# show diff ?
rollback-patch
Show rollback patch between configuration files or checkpoints
SW1# show diff rollback-patch ?

checkpoint
Use checkpoint as source configuration
file
Src Checkpoint file
running-config Use running configuration as source
startup-config Use startup configuration as source
SW1# show diff rollback-patch checkpoint ?
CONFIG_BACKUP Checkpoint name
The available rollbacks are shown which you can use for the comparing.
SW1# show diff rollback-patch checkpoint CONFIG_BACKUP ?
checkpoint
Use checkpoint as destination configuration
file
Dst Checkpoint file
running-config Use running configuration as destination
startup-config Use startup configuration as destination
SW1# show diff rollback-patch checkpoint CONFIG_BACKUP startup-config ?
<CR>
>
>>
|
Pipe command output to filter
Then you should specify to which configuration the compare should be done. In this case we
are using the startup-config.
SW1# show diff rollback-patch checkpoint CONFIG_BACKUP startup-config
Collecting Startup-Config
#Generating Rollback Patch
.....
!!
no logging timestamp microseconds
no logging server 172.16.100.110 5 use-vrf management facility local3
no monitor session 2
no monitor session 1
no clock timezone CET 1 0
494

cdp enable
exit
cdp enable
exit
cdp enable
exit
cdp enable
exit
cdp enable
exit
cdp enable
exit
cdp enable
exit
cdp enable
exit
no switchport monitor
cdp enable
exit
no switchport monitor
no switchport mode trunk
cdp enable
exit
cdp enable
exit
no interface port-channel1011
no interface port-channel110
no cdp format device-id serial-number
no cdp timer 10
no vlan 601
no ntp trusted-key 1
no ntp authentication-key 1 md5 WeqjAPS 7
no ntp authenticate
no ntp server 198.18.99.1
no snmp-server community IPexpert group network-operator
no snmp-server host 172.16.100.110 traps version 2c IPexpert
no snmp-server globalEnforcePriv
no snmp-server location Utrecht, Netherlands, Europe
no snmp-server contact Rick Mur, IPexpert
no ip access-list NTP_CLIENTS
no username version3 role storage-admin
no username version3
no username readonly
no role name storage-admin
!
interface vfc1
switchport mode E
no shutdown
!
logging monitor
SW1#
495
The configuration output that is shown then includes a patch that you can use as a
configuration template. The configuration is made so when it is applied to the current
configuration (the one you are comparing with). Then it would rollback to that configuration.
Its a very useful feature before doing changes, to automatically create your rollback script.
The script also ensures that commands are typed in the correct order, so you can easily copy
and paste it to the device when there are problems with your change or your current
configuration.
The next questions states that the configuration should be saved to a TFTP server on a weekly
basis on Sunday night at 10PM or 22:00.
The file-name should include some variables like the hostname and the date and time.
For this feature we are going to use a simple copy command, but we need to schedule it.
NX-OS knows a scheduler feature in which you are able to schedule CLI commands and also
use different variables to ensure the command executed includes things like the hostname
or the date and time.
We configure the scheduler job first, containing the commands that need to be executed.
SW1
SW1(config)# feature scheduler
SW1(config)# scheduler ?
aaa-authentication Password for AAA authentication(of logged in user)
job
Define a job
logfile
Scheduler log file configuration
schedule
Define a schedule
transport
Configure transport related configuration
SW1(config)# scheduler job ?
name Specify a name for the job
SW1(config)# scheduler job name SAVE_CONFIG
SW1(config-job)# copy run tftp://172.16.100.103/config-$(SWITCHNAME)$(TIMESTAMP).txt
SW1(config-job)# exit
Within the scheduler job you have access to the full CLI. This CLI access gives you the
possibility to change anything in the configuration or any show commands necessary.
Ensure you are using the correct variables in the job. These can be found in the Cisco
Documentation. For now the hostname and timestamp are the only variables available in the
scheduler, but new ones can be added in newer versions of software.
SW1
SW1(config)# scheduler schedule name CONFIG_BACKUP_SCHED
SW1(config-schedule)# job name SAVE_CONFIG
SW1(config-schedule)# ?
496

email-addr
job
no
time
end
exit
pop
push
where
Add email addr to send output of jobs configured for this schedule
Assign a job to the schedule
Specify a schedule to run jobs
Go to exec mode
SW1(config-schedule)# time ?
daily
Specify a daily schedule
monthly Specify a monthly schedule
start
Specify a future time schedule
weekly
Specify a weekly schedule
SW1(config-schedule)# time weekly ?
WORD Day-of-week {[[dow:]HH:]MM} dow = 1 (Sun) .. 7 (Sat) (Max Size 10)
SW1(config-schedule)# time weekly Sun:22:00
SW1(config-schedule)# exit
After the creation of the job, a scheduler needs to be created. This finalizes the
configuration and ensures that the selected job is run at the given time and date.
We can verify if the job was executed successfully and what results where. When errors occur
during the execution of scheduler jobs. These are logged in a logfile.
Its also possible to configure a username and password which is used for executing the jobs so
the accounting records are filled as well.
By default the admin user is used as the user to execute the jobs.
SW1(config)# show scheduler job
Job Name: SAVE_CONFIG
--------------------copy running-config tftp://172.16.100.103/config-$(SWITCHNAME)-$(TIMESTAMP).txt
==============================================================================
N7K11-SW1(config)# show scheduler schedule
Schedule Name
: CONFIG_BACKUP_SCHED
----------------------------------------User Name
: admin
Schedule Type
: Run on every Sunday at 22 Hrs 0 Mins
Last Execution Time : Yet to be executed
----------------------------------------------Job Name
Last Execution Status
----------------------------------------------SAVE_CONFIG
-NA==============================================================================
SW1(config)#
497
Now the schedule job is scheduled and the configuration is backed up to a TFTP server every
week.
The next question is related to a banner that should be shown to users when they log in. As
with classical IOS. This is done using the Message Of The Day (MOTD) banner.
SW1
SW1# conf t
SW1(config)# banner ?
motd Configure banner motd message
End with CNTL/Z.
SW1(config)# banner motd "IPexpert CCIE Data Center Lab"

SW1(config)# end
SW1# exit
Connection to 172.16.100.1 closed.
RetinaRick:~ rick$ ssh 172.16.100.1
IPexpert CCIE Data Center Lab
SW1#
After configuring the banner it is shown when users are logging in to the device.
The last questions are related to the archiving features of the NX-OS platform. You are able to
save files and compress them to GZIP files. After which they can then be combined to TAR files.
This is especially good to do with tech-support files, as they are very lengthy and can
generate megabytes of clear text.
In the first question we need to save the tech-support and create a zip file manually.
SW1# show tech-support > bootflash:techsupport.txt
SW1# dir bootflash:
18623
Nov 23 11:24:39 2012 CONFIG_BACKUP_2
1256
Mar 18 16:33:53 2011 aaa_cnv.log
364
Mar 18 16:33:53 2011 assoc_mgr_cnv.log
505
Mar 21 17:26:12 2010 license_SSI140806X2_6.lic
49152
Mar 18 16:14:02 2011 lost+found/
3785
Nov 07 20:01:14 2012 mts.log
21778944
May 18 13:50:04 2010 n5000-uk9-kickstart.4.2.1.N1.1.bin
25164288
Feb 21 11:36:01 2011 n5000-uk9-kickstart.5.0.2.N2.1.bin
25152512
Mar 18 15:59:51 2011 n5000-uk9-kickstart.5.0.3.N1.1a.bin
181095489
May 18 13:46:13 2010 n5000-uk9.4.2.1.N1.1.bin
156932426
Feb 21 11:37:53 2011 n5000-uk9.5.0.2.N2.1.bin
188758273
Mar 18 16:02:21 2011 n5000-uk9.5.0.3.N1.1a.bin
4939579
Nov 23 12:55:41 2012 techsupport.txt
4096
Jan 01 01:02:43 2005 vdc_2/
498

4096
4096
Jan 01 01:02:43 2005

Jan 01 01:02:43 2005
vdc_3/
vdc_4/
Usage for bootflash://sup-local

After the output of the tech-support is saved to a text file on the flash. We see that the
size is almost 5 megabytes. Now we need to compress it.
SW1# gzip bootflash:techsupport.txt
SW1# dir bootflash:
18623
Nov 23 11:24:39 2012
1256
Mar 18 16:33:53 2011
364
Mar 18 16:33:53 2011
505
Mar 21 17:26:12 2010
49152
Mar 18 16:14:02 2011
3785
Nov 07 20:01:14 2012
21778944
May 18 13:50:04 2010
25164288
Feb 21 11:36:01 2011
25152512
Mar 18 15:59:51 2011
181095489
May 18 13:46:13 2010
156932426
Feb 21 11:37:53 2011
188758273
Mar 18 16:02:21 2011
347146
Nov 23 12:55:41 2012
4096
Jan 01 01:02:43 2005
4096
Jan 01 01:02:43 2005
4096
Jan 01 01:02:43 2005
CONFIG_BACKUP_2
aaa_cnv.log
assoc_mgr_cnv.log
license_SSI140806X2_6.lic
lost+found/
mts.log
n5000-uk9-kickstart.4.2.1.N1.1.bin
n5000-uk9-kickstart.5.0.3.N1.1a.bin
n5000-uk9.4.2.1.N1.1.bin
n5000-uk9.5.0.2.N2.1.bin
n5000-uk9.5.0.3.N1.1a.bin
techsupport.txt.gz
vdc_2/
vdc_3/
vdc_4/
Usage for bootflash://sup-local

SW1#
After compression the file is only 347 kilobytes.

In this question we compressed the file manually. Its also possible to create automatically
compressed files from the output.
SW1
SW1# terminal ?
alias
persistent).
color
error), command
dont-ask
edit-mode
event-manager
history
length
log-all
monitor
no
output
Show aliases (if no arugments); create 'exec' aliases (not

Persistent aliases are in config mode, see 'cli alias'
Enable colorization of prompt(green if last command ok, red if
line (blue), output (default color)
Don't ask 'are you sure' questions, take default answer instead
Set command line edition keys (vi or emacs; emacs is default)
Event manager cli event
Configure terminal history properties
Set number of lines on a screen
Accounting log all commands including the show commands
Copy Syslog output to the current terminal line
How output of show commands should be formated
499

redirection-mode
session-timeout
terminal-type
tree-update
verify-only
width
Set the redirection mode

Set session timeout
Set the terminal type
Updates the main parse tree
Verify command and do not execute
Set width of the display terminal
SW1# terminal redirection-mode ?

ascii
Set the redirection mode to ASCII
zipped Set the redirection mode to ZIPPED
SW1# terminal redirection-mode zipped
SW1#
SW1# show interface > interface.txt
With the redirection-mode configuration (the > command) we can set how files should be
saved. By default they are saved as standard ASCII files, but now we changed it to ZIP files.
500
While the file is still saved as .txt file. The output is unreadable as it is zipped.
SW1# dir
18623
1256
364
3101
505
49152
3785
21778944
25164288
25152512
181095489
156932426
188758273
347146
4096
4096
4096
Nov
Mar
Mar
Nov
Mar
Mar
Nov
May
Feb
Mar
May
Feb
Mar
Nov
Jan
Jan
Jan
23
18
18
23
21
18
07
18
21
18
18
21
18
23
01
01
01
11:24:39
16:33:53
16:33:53
12:59:27
17:26:12
16:14:02
20:01:14
13:50:04
11:36:01
15:59:51
13:46:13
11:37:53
16:02:21
12:55:41
01:02:43
01:02:43
01:02:43
2012
2011
2011
2012
2010
2011
2012
2010
2011
2011
2010
2011
2011
2012
2005
2005
2005
CONFIG_BACKUP_2
aaa_cnv.log
assoc_mgr_cnv.log
interface.txt
lost+found/
mts.log
n5000-uk9.4.2.1.N1.1.bin
n5000-uk9.5.0.2.N2.1.bin
n5000-uk9.5.0.3.N1.1a.bin
techsupport.txt.gz
vdc_2/
vdc_3/
vdc_4/

SW1# tail interface.txt
???.SH??K?22@)?pLc??,c?fV????7h??#k?`?rX??!h?p?T?^hh????=????`>J?$4??5pQ??q???D
?C
\?)4O*?cQ?He???`SV?M?{)?|??Hv??!0wrd-4J?
9?Rn4W
??Ak??v4?Z?&??l:W??$?'E?L\_???Y???????C?K???UJ?t?x?0???A?f?3?r??sd?W??X|V?a??B
G??6??FZ???w?T
??x???+Y????
sj?8w?.?l??f????vHN}??n?I??
? O| 2 #
#x;v?w?Z?jyZ?7
?i??4??y?!
+?#?%?A?\?!]Pc;~?@
???i{Z?O`K?K?5?|?>?`U??U ?;??Pdk:2?F6??F?????>w ???R|l?}??|z?zK?\?h{K?
&??zK4S?p?
?&??z??(i,????YI?(-a?F
?oKK&???-??b???????
?oh>
F????q?}??{?;?lpOl,?f????????? ?yi??&@??????Z?8?-???v?{?k??
????^?Fz?n`t
?3Z???`??)? ?\KT??I?T?`??B???
?1???WJ???????Z?w??*\|x???????+???.}??~I/???/S?8?Ee?V?+??z??k#??+????:??
?Vc?&?fL?[??m??J+??5?H?[u.?T?R3#?;??
?Q?j
???U[B1j?g?
Z}?Zo???g?=%u,?j??s
9v?v??
SW1# ?R?
As the output above shows the file is unreadable unless unzipped.

The final question states that both files that are generated should be combined in a TAR file.
SW1
SW1# tar ?
501
append
create
extract
list
Append some files to an existing archive

Create an archive (merge several files together)
Extract files from archive (unmerge them)
Shows the list of files which are part of the archive
SW1# tar create ?

bootflash: The name of the archive (extension will be added if none of
tar/tgz/tar.gz/tar.bz2/tbz2/tar.Z specified)
volatile:
The name of the archive (extension will be added if none of
tar/tgz/tar.gz/tar.bz2/tbz2/tar.Z specified)
SW1# tar create
absolute
bootflash:
bz2-compress
gz-compress
remove
uncompressed
verbose
volatile:
bootflash:combined ?
Don't strip leading `/'s from file names
Name of file to be added into archive
Compress archive with bzip2 -> .tar.bz2
Compress archive with gzip, the default -> .tar.gz
Remove files after adding them to the archive
Dont compress archive -> .tar
Display files while merging/extracting
Name of file to be added into archive
SW1# tar create bootflash:combined bootflash:interface.txt

bootflash:techsupport.txt.gz
SW1#
We first specify the name of the archive and then specify which files should be included in
the archive.
SW1#
SW1# dir
18623
1256
364
330925
3101
505
49152
3785
21778944
25164288
25152512
181095489
156932426
188758273
347146
4096
4096
4096
Nov
Mar
Mar
Nov
Nov
Mar
Mar
Nov
May
Feb
Mar
May
Feb
Mar
Nov
Jan
Jan
Jan
23
18
18
23
23
21
18
07
18
21
18
18
21
18
23
01
01
01
11:24:39
16:33:53
16:33:53
13:05:32
12:59:27
17:26:12
16:14:02
20:01:14
13:50:04
11:36:01
15:59:51
13:46:13
11:37:53
16:02:21
12:55:41
01:02:43
01:02:43
01:02:43
2012
2011
2011
2012
2012
2010
2011
2012
2010
2011
2011
2010
2011
2011
2012
2005
2005
2005
CONFIG_BACKUP_2
aaa_cnv.log
assoc_mgr_cnv.log
combined.tar.gz
interface.txt
lost+found/
mts.log
n5000-uk9.4.2.1.N1.1.bin
n5000-uk9.5.0.2.N2.1.bin
n5000-uk9.5.0.3.N1.1a.bin
techsupport.txt.gz
vdc_2/
vdc_3/
vdc_4/

SW1# tar list bootflash:combined.tar.gz
interface.txt
techsupport.txt.gz
SW1#
502
We finally see the file is created and that the archive contains the correct files just like we
wanted.
Just like the question specified, the TAR file should be compressed. Which is the default when
not specifying anything about the archive in the CLI.
This finished the task about the system management!
Task 6: Smart Call Home and GOLD
The final task is about configuring the possibility of the device to report critical issues to an
operator or even to Cisco TAC. This feature is called CallHome or Smart Call Home.
Smart Call Home offers the possibility to send critical information about the Nexus device to
either an operator or on-call engineer. The other option is that the feature will analyze the
information and file a request with the correct Cisco TAC team.
When your Nexus is not connected to an internet connection. So a connection to a mail server
or to Cisco TAC is not possible. You are able to use a Transport Gateway (TG) aggregation
point. This TG is connected to the internet and connected to the internal management network
to ensure a secure connection between the Nexus device and Cisco TAC.
Its possible to distribute Callhome configuration through CFS between all Nexus and MDS
switches in the network.
Once a Call Home message is generated you are able to configure what kind of information
should be shared with Cisco. This can come from default pre-sets of configurations or you can
enter certain CLI commands to be entered to collect more information.
Then there is an option to send the messages in brief text messages, full text messages
or XML messages. The XML messages are only used for filing TAC cases.
The pre-sets that are available for a default set of commands to be reported are called AlertGroups. The following table explains which alert-groups execute which command to use in
the alert.
Alert-Group
Description
Commands
Cisco-TAC
All critical alerts from the other

alert groups destined for Smart
Call Home.
Execute commands based on the alert

group that originates the alert.
503
Diagnostic
Events generated by
diagnostics.
show diagnostic result module all

detail
show moduleshow version
show tech-support platform callhome
Supervisor
hardware
Events related to supervisor

modules.

detail
Linecard
hardware
Events related to standard or

intelligent switching modules.

detail
Configuration
Periodic events related to

configuration.
show version
show module
show running-config all
show startup-config
System
Events generated by failure of a show system redundancy status

software system that is critical show tech-support
to unit operation.
Environmental
Events related to power, fan,

and environment-sensing
elements such as temperature
alarms.
show environment
show logging last 1000
show module show version
Inventory
Inventory status that is provided

whenever a unit is cold booted,
or when linecards are inserted
or removed. This alert is
considered a noncritical event,
and the information is used for
status and entitlement.
show module
show version
show license usage
show inventory
show sprom all
show system uptime
504
Configuration of the Call-Home feature requires a number of components.

You first need to configure SNMP-Server Contact information. Without this configured
CallHome will not work! This is also the only option which is not distributed with CallHome.
You need to manually configure the SNMP server contact information before CallHome actually
works.
We start our configuration with a boot-up configuration. The feature that is intended here is
called GOLD. GOLD ensures that the right amount of diagnostics are collected. By default all
notifications are enabled, so the only option to configure is in this case the amount of
diagnostics that should be collected during the boot process of the switch.
SW1
SW1(config)# diagnostic ?
bootup Configure Diagnostic for bootup
SW1(config)# diagnostic bootup ?
level Select diagnostic level
SW1(config)# diagnostic bootup level ?
bypass
Skip all bootup test
complete Complete level
extra
Extra level
SW1(config)# diagnostic bootup level complete
SW2
SW3
The results can be made visible with a show diagnostic results command.
Next we can start by configuring the Call Home feature.
We do not need to enable the feature, because its always enabled.
SW1
SW1(config)# feature call
^
SW1(config)#
SW1(config)#
SW1(config)# snmp-server contact Rick Mur, IPexpert
505
As explained, the SNMP contact (SysContact) information should be supplied, before

CallHome will work.
SW1
SW1(config)# callhome
SW1(config-callhome)# ?
alert-group
Alert group
contract-id
Service contract id of the customer
customer-id
Customer id
destination-profile Configure destination profiles
duplicate-message
Configure throttling of duplicate callhome alert messages
email-contact
Email address of the contact person
enable
Enable callhome
no
periodic-inventory
Configure periodic software inventory message dispatch
phone-contact
Contact person's phone number
site-id
Site id of the network where switch is deployed
streetaddress
Configure replacement part shipping address.
switch-priority
Priority of the switch(0-highest 7-lowest)
transport
Configure transport related configuration
end
Go to exec mode
exit
pop
push
where
SW1(config-callhome)# alert-group ?
Configuration
Events related to Configuration
Diagnostic
Events related to Diagnostic
Environmental
Power,fan,temperature related events
Inventory
Inventory status events
License
Events related to licensing
Linecard-Hardware
Linecard related events
Supervisor-Hardware Supervisor related events
Syslog-group-port
Events related to syslog messages filed by port manager
System
Software related events
Test
User generated test events
Alert-groups can be configured to be expanded with several CLI commands.

SW1
SW1(config-callhome)# destination-profile ?
CiscoTAC-1
Configure destination profile for XML message
WORD
User defined destination profile name (Max Size 31)
full-txt-destination
Configure destination profile for plain txt message
short-txt-destination Configure destination profile for short txt message
We are configuring a destination-profile, just like our questions state.

We should be configuring a profile that sends messages to an e-mail address and specifying all
messages to be sent.
SW1
SW1(config-callhome)# destination-profile CRITICAL_MAILS ?
<CR>
506
alert-group
email-addr
format
http
message-level
message-size
transport-method
Add alert group

Add email addr
Callhome message format (default XML)
Add http or https url
Callhome message level(0-lowest urgency, 9-highest urgency)
Configure maximum message size (default 2500000)
Callhome message sending transport-method
SW1(config-callhome)# destination-profile CRITICAL_MAILS email-addr ?

WORD Provide email address, example: jdow@xyz.com (Max Size 255)
SW1(config-callhome)# destination-profile CRITICAL_MAILS
SW1(config-callhome)# destination-profile CRITICAL_MAILS email-addr
callhome@ciscocallhome.com
<CR>
alert-group
Add alert group
email-addr
Add email addr
format
http
message-level
message-size
transport-method Callhome message sending transport-method
SW1(config-callhome)# destination-profile CRITICAL_MAILS transport-method ?
email Email transport-method
http
Http transport-method
SW1(config-callhome)# destination-profile CRITICAL_MAILS transport-method email
Now the destination e-mail address and method are specified. Next up is configuring the
source.
SW1
SW1(config-callhome)# email-contact ?
WORD Provide email address, example: jdow@xyz.com (Max Size 255)
SW1(config-callhome)# email-contact rmur@ipexpert.com ?
SW1(config-callhome)# streetaddress Cisco Street 111
The street address should also be specified as that is used as a field in the Callhome message.
SW1
SW1(config-callhome)# transport ?
email Configure email transport related configuration
SW1(config-callhome)# transport email
from
reply-to
smtp-server
SW1(config-callhome)# transport email ?
from
Configure from email address
reply-to
Configure replyto email address
smtp-server Configure SMTP server address
SW1(config-callhome)# transport email from rmur@ipexpert.com
SW1(config-callhome)# transport email from sm?
WORD Provide from email address, example: SJ-9500-1@xyz.com (Max Size 255)
SW1(config-callhome)# transport email smtp-server ?
A.B.C.D
IPV4 address of SMTP server
507
A:B::C:D
WORD
IPV6 address of SMTP server

SMTP server(DNS name or IPv4 or IPv6 address)
SW1(config-callhome)# transport email smtp-server mail.ciscocallhome.com ?

<CR>
port
Configure SMTP server port (default:25)
use-vrf Configure vrf name
SW1(config-callhome)# transport email smtp-server mail.ciscocallhome.com
SW1(config-callhome)# exit
We configure the SMTP server and the source e-mail address that should be used in our
configuration. Pay attention that you are only able to configure 1 source address and SMTP
server, but multiple destination addresses. The destination address is configured under the
destination-profile.
SW1
SW1(config)#
Cannot write
SW1(config)#
SW1(config)#
ip
to
ip
ip
name-server 172.16.100.111
the dns configuration -DNS is disabled
domain-lookup
name-server 172.16.100.111
Because we are configuring the name of the SMTP server in our configuration, we should use
DNS name resolution to ensure that the IP address of the mail server is resolved.
Pay attention to enable name-resolution globally.
SW1
SW1(config)# callhome
SW1(config-callhome)#
CiscoTAC-1
WORD
<CR>
alert-group
Add alert group
email-addr
Add email addr
format
http
message-level
message-size
SW1(config-callhome)# destination-profile CRITICAL_MAILS mess
message-level
message-size
SW1(config-callhome)# destination-profile CRITICAL_MAILS message-level ?
<0-9>
SW1(config-callhome)# destination-profile CRITICAL_MAILS message-level 0
SW1(config-callhome)# destination-profile CRITICAL_MAILS message-size ?
<0-5000000> Provide maximum possible message size
508
SW1(config-callhome)# destination-profile CRITICAL_MAILS message-size 5000000

SW1(config-callhome)# destination-profile CRITICAL_MAILS format ?
XML
XML message format
full-txt
Plain text message format
short-txt Short text message format
SW1(config-callhome)# destination-profile CRITICAL_MAILS format full-txt
Finally we should specify the maximum message size. We want to sent all messages of any
urgency therefore we configured the level 0 urgency.
Next we should configure periodic inventory reports, which is another feature of CallHome.
SW1
SW1(config-callhome)# periodic-inventory ?
notification Enable periodic software inventory message dispatch
SW1(config-callhome)# periodic-inventory notification ?
<CR>
interval
Configure the time period for periodic inventory
timeofday Configure the timeofday for periodic inventory in HH:MM format
SW1(config-callhome)# periodic-inventory notification interval ?
<1-30> Time period in days (default is 7 days)
SW1(config-callhome)# periodic-inventory notification interval 1
SW1(config-callhome)# periodic-inventory ?
notification Enable periodic software inventory message dispatch
SW1(config-callhome)# periodic-inventory notification ?
<CR>
interval
Configure the time period for periodic inventory
timeofday Configure the timeofday for periodic inventory in HH:MM format
SW1(config-callhome)# periodic-inventory notification
The next question specifies that SW1 is a very important switch which should be noticed in the
CallHome configuration. This is specified through the priority configuration.
SW1
SW1(config-callhome)# switch-priority ?
<0-7> Priority of the switch(0-highest 7-lowest)
SW1(config-callhome)# switch-priority 0
509
Finally we should ensure that Cisco TAC receives information, both via e-mail and HTTP
requests.
This is configured with XML destination-profiles. However we can only configure one
method per destination-profile. We can only configure one additional profile, so we
need to use one predefined profile.
The pre-defined profiles are configured with a set of alert-groups, but also give you room
for additional configuration, like the transport-methods.
We first configure the HTTP version using the pre-defined Cisco TAC profile.
SW1
SW1(config-callhome)# callhome
CiscoTAC-1
WORD
SW1(config-callhome)# destination-profile CiscoTAC-1 ?
alert-group
Add alert group
email-addr
Add email addr
http
SW1(config-callhome)# destination-profile CiscoTAC-1 transport-method http
Now we used the pre-defined TAC profile for sending the HTTP requests. As you can see the
configuration is limited of this profile, because it uses a number of defaults.
Next we can finish the configuration of this chapter with configuring an additional
destination-profile for sending e-mails.
SW1
SW1(config-callhome)# destination-profile CiscoTAC-2 transport-method email
SW1(config-callhome)# destination-profile CiscoTAC-2 email-addr
ciscotac@ciscocallhome.com
SW1(config-callhome)# destination-profile CiscoTAC-2 ?
<CR>
alert-group
Add alert group
email-addr
Add email addr
format
http
message-level
message-size
SW1(config-callhome)# destination-profile CiscoTAC-2 format xml
SW1(config-callhome)# destination-profile CiscoTAC-2
510
Now every destination-profile is correctly configured so the final task is to enable the
callhome configuration.
SW1(config-callhome)# enable
contact phone is not configured
callhome cannot be enabled on the switch,
because necessary configuration has not been done
Please check if all of following configuration is done
contact person name(sysContact)
contact person's email
contact person's phone number
street addr
To configure sysContact, please use snmp-server command
SW1(config-callhome)# phone-contact +111333444555666
SW1(config-callhome)# enable
The enabling of Callhome will warn you when something has been forgotten. In this case we
forgot to configure the phone number. After the phone number has been configured
Now we finished the configuration. Lets verify if everything is working as expected.
SW1(config-callhome)# show run callhome
!Command: show running-config callhome
!Time: Sun Nov 25 17:45:52 2012
callhome
switch-priority 0
email-contact rmur@ipexpert.com
phone-contact +111333444555666
streetaddress Cisco Street 111
destination-profile CRITICAL_MAILS
destination-profile CRITICAL_MAILS format full-txt
destination-profile CiscoTAC-1 transport-method http
destination-profile CiscoTAC-2
destination-profile CiscoTAC-2 format XML
destination-profile CRITICAL_MAILS email-addr callhome@ciscocallhome.com
destination-profile CiscoTAC-1 email-addr ciscotac@ciscocallhome.com
destination-profile CiscoTAC-2 email-addr ciscotac@ciscocallhome.com
transport email smtp-server mail.ciscocallhome.com port 25
transport email from rmur@ipexpert.com
enable
periodic-inventory notification interval 1
SW1(config-callhome)# show callhome

callhome enabled
Callhome Information:
contact person name(sysContact):Rick Mur, IPexpert
contact person's email:rmur@ipexpert.com
contact person's phone number:+111333444555666
street addr:Cisco Street 111
site id:
customer id:
contract id:
switch priority:0
511
duplicate message throttling : enabled

periodic inventory : enabled
periodic inventory time-period : 1 day
periodic inventory timeofday : 08:00 (HH:MM)
Distribution : Disabled
SW1(config-callhome)# show callhome destination-profile
full_txt destination profile information
maximum message size:2500000
message format:full-txt
message-level:0
transport-method:email
email addresses configured:
url addresses configured:
alert groups configured:
all
short_txt destination profile information
message format:short-txt
message-level:0
all
CiscoTAC-1 destination profile information
message format:XML
message-level:0
transport-method:email and http
cisco-tac
CiscoTAC-2 destination profile information

message format:XML
message-level:0
CRITICAL_MAILS destination profile information

message format:full-txt
message-level:0
callhome@ciscocallhome.com
512

After verification we finished the task and we finished the chapter.

This chapter concluded the configuration chapters of NX-OS equipment. The next chapter will
all be about configuring the Cisco UCS system. This has been divided in several chapters related
to Networking, Storage and Compute.
513

IPexpert CCIE Data Center Volume 1 Detailed Solution Chapters 1 8 PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IPexpert CCIE Data Center Volume 1 Detailed Solution Chapters 1 8 PDF

Uploaded by

Copyright:

Available Formats

IPexperts Lab Preparation Detailed Solution Guide

for the Cisco CCIE Data Center v1.0 Lab Exam

CCIE Data Center Lab Preparation Detailed Solution Guide

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Additional CCIETM Preparation Material

Issues with this Book

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

IPEXPERT END-USER LICENSE AGREEMENT

Copyright and Proprietary Rights

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Choice of Law and Jurisdiction

Limitation of Claims and Liability

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

U.S. Government - Restricted Rights

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

CCIE Data Center Lab Preparation Detailed Solution Guide

Task 1: Layer 3 topology set-up .......................................................................................................... 66

CCIE Data Center Lab Preparation Detailed Solution Guide

Task 1: Port Security ......................................................................................................................... 395

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Default Lab Topology

Default passwords and IP addresses

Default management username / password: admin / IPexpert123

Management IP addressing: 172.16.100.0/24

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Who Should Read this Book?

How to Use this Book

An Introduction to CCIE Data Center

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

UCS B-series blade systems

The live exam is available from September 1st.

The current published reading list:

NX-OS and Cisco Nexus Switching (ISBN-10: 1-58705-892-8)

Cisco Unified Computing System (UCS) (ISBN-10: 1-58714-193-0)

I/O Consolidation in the Data Center (ISBN-10: 1-58705-888-X)

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Storage Networking Fundamentals (ISBN-10: 1-58705-162-1)

NXOS v6.0(2) on Nexus 7000 Switches

What about P and A tracks?

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

An Introduction to the Proctor Labs CCIE Data Center hardware rack

The blueprint specified the following components to be in the lab:

First is the NX-OS Networking equipment.

Nexus7009 (with licensing)

Copyright by IPexpert. All rights reserved.

CCIE Data Center Lab Preparation Detailed Solution Guide

Second is the storage networking (SAN) equipment:

Dual attached JBODs = Fibre Channel disks

CCIE Data Center Lab Preparation Detailed Solution Guide

UCS-6248 Fabric Interconnects