Professional Documents
Culture Documents
An Introduction
Authors: Garry Lemasa and Silvano Gai, Cisco Systems
Introduction is a virtualization platform that allows multiple
Fibre Channel over Ethernet (FCoE) is a newly operating systems to run on a host computer at
proposed standard that is being developed by the same time.) Companies who have virtual-
INCITS T11. The FCoE protocol specification machine based mission critical applications in
maps Fibre Channel natively over Ethernet and is their data-centers typically configure their servers
independent of the Ethernet forwarding scheme. with a pair of Fibre Channel host bus adapters
It allows an evolutionary approach towards I/O (HBA’s) and two or more network interface cards
consolidation by preserving all Fibre Channel (NICs). Some high end deployments have as
constructs, maintaining the same latency, security, many as eight Ethernet NICs in high performance
and traffic management attributes of FC while servers that only have 2 CPU cores! The adapter
preserving investments in FC tools, training, and requirements could only grow when 32 CPU
SANs. FCoE recognizes FC as the dominant storage cores become commonplace. Fibre Channel over
protocol in the data center while giving customers Ethernet (FCoE) enables the consolidation of both
a viable I/O consolidation solution. FCoE simplifies SANs and Ethernet traffic onto a one common
customer environments by using Ethernet and network adapter, reducing the ever growing
allowing the industry to avoid creating another, number of adapters required.
separate protocol for I/O consolidation. FCoE combined with the advent of 10 Gigabit
In today’s datacenters, companies use both Ethernet (10 GE) fabrics will grant companies
Ethernet for TCP/IP networks and Fibre Channel the ability to consolidate their I/O, cables, and
(FC) for storage area networks (SANs), each adapters while at the same time increase the
dedicated to specific purposes. Ethernet networks utilization of their servers.
are generally implemented when end-users need The current proposal, as defined by the INCITS
to transfer relatively small amounts of information T11 standards body, leverages a lossless Ethernet
over both local and global distances or in clustered, fabric, maintains the Fibre Channel operational
low-latency compute environments. Storage area model and includes a newly approved frame
networks are implemented by companies who format. FCoE is not tied to 10GE, and will be
require access to block I/O for applications such as able to run over networks with varying interface
booting over SANs, mail servers, file servers and speeds.
large databases. The benefits of deploying SANs
remain: A) centralized management, security, Why is FCoE important to the data-center?
and administration of the storage resources, B)
I/O Consolidation.
uniform delivery of storage services like periodic
backups, and C) running efficient utilization levels I/O consolidation is simple in concept: the sharing
of storage resources. of both Fibre Channel and Ethernet traffic on
the same physical cable or in cases that network
Server virtualization is on the rise in data centers
isolation is desired, the flexibility to use and
as well, spurring an increased demand for Fibre
configure the same hardware for either type of
Channel connectivity to virtual hosts residing
network load. The benefits end-users will realize
on these servers. Demand for Fibre Channel in
from this simple idea are significant. Companies
this case is driven by the need for hypervisors
that leverage I/O consolidation will be able to
to provide guest operating systems with
realize significant gains in server slot efficiencies
virtualized storage, accessible through a Fibre
with the use of multi-function network/storage
Channel network infrastructure. (A hypervisor
www.fibrechannel.org
Fibre Channel over Ethernet in the Data Center: An Introduction
adapters simplify their cabling scheme within a those servers generate. In a standard PCI Express
rack and reduce the amount of heat each server bus, each adapter has a power budget of 25 watts.
produces. Even better, companies will be able to If an IT manager reduced the number of adapters
accomplish these things while still utilizing their in the rack server from six to two, they would be
existing Fibre Channel SANs. These benefits are potentially saving up to 100 watts per rack server
further detailed below. (25 watts/ adapter X 4 adapters = 100 Watts).
Today, IT managers often use four, six or even I/O consolidation also gives companies the
eight network adapters in their mission critical means to simplify their cable management.
servers. These adapters can be two Fibre Channel Implementing FCoE on a pair of 10 Gigabit
host bus adapters, plus a pair of Network Ethernet cables can replace the equivalent of
Interface Cards for traffic and in the case of a two 4 Gb FC connections and twelve one Gigabit
Virtual Machine environment up to an additional Ethernet connections. While still providing 20
four NICs, depending on the VM vendor’s best Gb of bandwidth, this is an 86% reduction in the
amount of cables to be managed.
For administrators that desire SAN/LAN isolation,
FCoE allows the user to employ only FCoE multi-
function network/storage adapters and without
the need to employ separate hardware and cables
based upon traffic types.
What is Fibre Channel over Ethernet?
Conceptually FCoE can be broken down into three
Figure 1 – FCoE I/O Consolidation Benefit components:
– Fewer NICs per Server • Encapsulation of a Native Fibre Channel Frame
into an Ethernet frame
practices. I/O consolidation means a customer
• The extension of Ethernet to become a lossless
can use multi-function network/storage adapters
fabric
in place of single-function network-specific and • The replacing of a Fibre Channel link with MAC
storage-specific cards, thereby reducing the
addresses in a lossless Ethernet fabric
number of server slots and switch ports, as well as
reducing the number of power consumed for I/O Why is preservation of Native Fibre Channel
and necessary cooling. This also results in fewer Format so Important?
points of management administrators will have to • Administration, Management and Service
control. Delivery
A reduction in NICs through I/O consolidation has Users contemplating deployment of an FCoE based
an additional important advantage. The ability network are assured of a smooth transition to
to cool a set amount of heat generated per rack is FCoE because existing Fibre Channel SAN based
the primary barrier to data-center expansion and administrative and management procedures
inefficiency encountered today. If a given rack has will continue to work in an FCoE based network.
a maximum of 12 kilowatts allotted for cooling, Compatibility with existing Fibre Channel
and that allotment is exceeded, then additional networks is required which will allow usage of
servers and switches cannot be added, regardless existing FC SAN tools and management constructs.
of the rack space available. Reducing the amount From a Storage Administrator perspective, zoning
of NICs in servers can reduce the amount of heat is a basic provisioning function that is used to give
Hosts access to Storage. Zoning has a dual purpose Finally, storage users can also be assured that
function A) storage allocation and B) security. By new services utilizing storage virtualization
virtue of zoning a Host is able to see the devices applications can continue to work with FCoE
provisioned for it on the Storage subsystem. because everything above the transport layer
Zoning also ensures that unauthorized servers remains intact. By encapsulating native Fibre
don’t get access to Storage devices. FCoE based Channel in FCoE, the transparent interoperability
switches will continue to offer this zoning function of existing SAN networks will be possible.
ensuring that storage allocation and security
mechanisms are unaffected. Encapsulation
In addition to zoning, another integral element in The encapsulation of the Fibre Channel frame
SANs is the Name Server Database. This database occurs through the mapping of FC onto Ethernet.
covers a list of devices currently connected to Fibre Channel and traditional networks have stacks
the network. It also includes information about of layers where each layer in the stack represents
network location and status of the connections. a set of functionality. The Fibre Channel stack
FCoE based SANs will continue to offer the Name consists of five layers, FC-0 through FC-4. Ethernet
Server Database function. is typically considered a set of protocols in the
seven-layer OSI stack that define the physical and
Beyond the basic SAN provisioning functions data link layers. FCoE provides the capability to
mentioned above, Storage Administrators carry the FC-2 layer over the Ethernet layer. See
also deliver services such as Disaster Recovery, figure 2 below. This allows Ethernet to transmit
Backup and Recovery. Disaster Recovery solutions the upper Fibre Channel layers FC-3 and FC-4 over
typically used a protocol called FCIP. FCIP remains the IEEE 802.3 Ethernet layers. .
unchanged with FCoE. Data that needs to be sent
to a remote site will hop on to a FCIP link via a It is this FCoE mapping which allows FC traffic to
gateway and similarly enter pass over an Ethernet infrastructure.
a FCoE network via a FCIP
gateway. File Backups to
Tape Libraries and Recovery
from these devices will
continue to use existing
backup and recovery tools.
What about companies
who have an existing Fibre
Channel infrastructure?
Does FCoE mean that
they need to replace their Figure 2 – FCoE mapping
existing SANs? Absolutely
not. FCoE is designed to maintain the Fibre
Channel model. A storage administrator logged
into a server connected through an FCoE network
would see Fibre Channel initiators just as if that
same server were connected on a SAN. The same
management tools that customers use to manage
and maintain their SANs today can be used in an
FCoE environment.
There is also a new Ethernet enhancement being Bridge where it is distributed to the appropriate
developed that allows PAUSE functionality to Ethernet port on the bottom of the diagram.
be enabled for each user-priority supported
in Ethernet. While PAUSE provides the basic The reverse is also true; FCoE traffic will flow from
functionality to make Ethernet lossless, a new the red ports, through the Ethernet Bridge until it
proposal for Priority Flow Control will provide is de-encapsulated inside of the FCoE Entity, where
significant benefit for both FC and Ethernet traffic. it passes through the FC switch to the appropriate
Priority Flow Control is discussed on page 7. FC port.
Example of an FCoE switch This is just one example of how a switch can be
FCoE enabled. Different vendors may choose to
Figure 5 shows a functional Diagram of a FCoE implement FCoE in different ways.
capable switch as depicted by the larger, yellow
field. Inside this device is both an Ethernet FC and FCoE address mapping
switch function as well as a Fibre Channel A third component of FCoE mentioned above is
switch function. Note the Fibre Channel over replacing of a Fibre Channel link MAC addresses
Ethernet entity within the FC Switch. This entity in a lossless Ethernet fabric. Traditional Fibre
encapsulates FC frames into FCoE Frames and de- Channel fabric switches maintain forwarding
encapsulates FCoE frames back to FC frames. FC tables FC_IDs. FC switches use these forwarding
traffic flows into one end of the FCoE entity and tables to select the best link available for a frame
FCoE traffic flows out the other end. so the frame reaches its destination port. Fibre
The FCoE Entity has an Ethernet MAC address that Channel links are typically point-to-point and do
will be used as a source or destination address not need an address at the link layer. An Ethernet
when FCoE frames are transmitted through an network is different because it does not form an
Ethernet fabric (Further details provided below). end-point to end-point connection in the manner
of FC. This requires FCoE to rely on Ethernet MAC
In figure 5, an FC frame enters from the green FC addresses to direct a frame to its correct Ethernet
ports to the right and into the FC switch where the destination (the first two fields of the FCoE frame
entity encapsulates the Fibre Channel frame into a are MAC addresses, as outlined previously in figure
FCoE frame. The FCoE frame is sent to an Ethernet 3).
Conclusion
Fibre Channel over Ethernet will allow companies’
data centers to optimize increases in server
virtualization for the next several years. FCoE
enables I/O consolidation and offers these
enhancements to the data-center:
• Fewer Network Interface Cards per server
• Reduction in server power budget and reduced
cooling requirements
• Significant reduction in the amount of cables
to manage
• Seamless connectivity with existing Storage
Area Networks and Ethernet Networks –
customers can continue to leverage their existing
FC tools, security model and infrastructure
FCoE positions Fibre Channel as the storage
networking protocol of choice and extends the
reach of Fibre Channel throughout the data center
to all servers.
Readers interested in learning more about FCoE
can refer to the following sources:
www.t11.org/FCoE
www.fibrechannel.org