You are on page 1of 8

Fibre Channel over Ethernet in the Data Center:

An Introduction
Authors: Garry Lemasa and Silvano Gai, Cisco Systems
Introduction is a virtualization platform that allows multiple
Fibre Channel over Ethernet (FCoE) is a newly operating systems to run on a host computer at
proposed standard that is being developed by the same time.) Companies who have virtual-
INCITS T11. The FCoE protocol specification machine based mission critical applications in
maps Fibre Channel natively over Ethernet and is their data-centers typically configure their servers
independent of the Ethernet forwarding scheme. with a pair of Fibre Channel host bus adapters
It allows an evolutionary approach towards I/O (HBA’s) and two or more network interface cards
consolidation by preserving all Fibre Channel (NICs). Some high end deployments have as
constructs, maintaining the same latency, security, many as eight Ethernet NICs in high performance
and traffic management attributes of FC while servers that only have 2 CPU cores! The adapter
preserving investments in FC tools, training, and requirements could only grow when 32 CPU
SANs. FCoE recognizes FC as the dominant storage cores become commonplace. Fibre Channel over
protocol in the data center while giving customers Ethernet (FCoE) enables the consolidation of both
a viable I/O consolidation solution. FCoE simplifies SANs and Ethernet traffic onto a one common
customer environments by using Ethernet and network adapter, reducing the ever growing
allowing the industry to avoid creating another, number of adapters required.
separate protocol for I/O consolidation. FCoE combined with the advent of 10 Gigabit
In today’s datacenters, companies use both Ethernet (10 GE) fabrics will grant companies
Ethernet for TCP/IP networks and Fibre Channel the ability to consolidate their I/O, cables, and
(FC) for storage area networks (SANs), each adapters while at the same time increase the
dedicated to specific purposes. Ethernet networks utilization of their servers.
are generally implemented when end-users need The current proposal, as defined by the INCITS
to transfer relatively small amounts of information T11 standards body, leverages a lossless Ethernet
over both local and global distances or in clustered, fabric, maintains the Fibre Channel operational
low-latency compute environments. Storage area model and includes a newly approved frame
networks are implemented by companies who format. FCoE is not tied to 10GE, and will be
require access to block I/O for applications such as able to run over networks with varying interface
booting over SANs, mail servers, file servers and speeds.
large databases. The benefits of deploying SANs
remain: A) centralized management, security, Why is FCoE important to the data-center?
and administration of the storage resources, B)
I/O Consolidation.
uniform delivery of storage services like periodic
backups, and C) running efficient utilization levels I/O consolidation is simple in concept: the sharing
of storage resources. of both Fibre Channel and Ethernet traffic on
the same physical cable or in cases that network
Server virtualization is on the rise in data centers
isolation is desired, the flexibility to use and
as well, spurring an increased demand for Fibre
configure the same hardware for either type of
Channel connectivity to virtual hosts residing
network load. The benefits end-users will realize
on these servers. Demand for Fibre Channel in
from this simple idea are significant. Companies
this case is driven by the need for hypervisors
that leverage I/O consolidation will be able to
to provide guest operating systems with
realize significant gains in server slot efficiencies
virtualized storage, accessible through a Fibre
with the use of multi-function network/storage
Channel network infrastructure. (A hypervisor

www.fibrechannel.org
Fibre Channel over Ethernet in the Data Center: An Introduction

adapters simplify their cabling scheme within a those servers generate. In a standard PCI Express
rack and reduce the amount of heat each server bus, each adapter has a power budget of 25 watts.
produces. Even better, companies will be able to If an IT manager reduced the number of adapters
accomplish these things while still utilizing their in the rack server from six to two, they would be
existing Fibre Channel SANs. These benefits are potentially saving up to 100 watts per rack server
further detailed below. (25 watts/ adapter X 4 adapters = 100 Watts).
Today, IT managers often use four, six or even I/O consolidation also gives companies the
eight network adapters in their mission critical means to simplify their cable management.
servers. These adapters can be two Fibre Channel Implementing FCoE on a pair of 10 Gigabit
host bus adapters, plus a pair of Network Ethernet cables can replace the equivalent of
Interface Cards for traffic and in the case of a two 4 Gb FC connections and twelve one Gigabit
Virtual Machine environment up to an additional Ethernet connections. While still providing 20
four NICs, depending on the VM vendor’s best Gb of bandwidth, this is an 86% reduction in the
amount of cables to be managed.
For administrators that desire SAN/LAN isolation,
FCoE allows the user to employ only FCoE multi-
function network/storage adapters and without
the need to employ separate hardware and cables
based upon traffic types.
What is Fibre Channel over Ethernet?
Conceptually FCoE can be broken down into three
Figure 1 – FCoE I/O Consolidation Benefit components:
– Fewer NICs per Server • Encapsulation of a Native Fibre Channel Frame
into an Ethernet frame
practices. I/O consolidation means a customer
• The extension of Ethernet to become a lossless
can use multi-function network/storage adapters
fabric
in place of single-function network-specific and • The replacing of a Fibre Channel link with MAC
storage-specific cards, thereby reducing the
addresses in a lossless Ethernet fabric
number of server slots and switch ports, as well as
reducing the number of power consumed for I/O Why is preservation of Native Fibre Channel
and necessary cooling. This also results in fewer Format so Important?
points of management administrators will have to • Administration, Management and Service
control. Delivery
A reduction in NICs through I/O consolidation has Users contemplating deployment of an FCoE based
an additional important advantage. The ability network are assured of a smooth transition to
to cool a set amount of heat generated per rack is FCoE because existing Fibre Channel SAN based
the primary barrier to data-center expansion and administrative and management procedures
inefficiency encountered today. If a given rack has will continue to work in an FCoE based network.
a maximum of 12 kilowatts allotted for cooling, Compatibility with existing Fibre Channel
and that allotment is exceeded, then additional networks is required which will allow usage of
servers and switches cannot be added, regardless existing FC SAN tools and management constructs.
of the rack space available. Reducing the amount From a Storage Administrator perspective, zoning
of NICs in servers can reduce the amount of heat is a basic provisioning function that is used to give

www.fibrechannel.org © 2007 Fibre Channel Industry Association


Fibre Channel over Ethernet in the Data Center: An Introduction

Hosts access to Storage. Zoning has a dual purpose Finally, storage users can also be assured that
function A) storage allocation and B) security. By new services utilizing storage virtualization
virtue of zoning a Host is able to see the devices applications can continue to work with FCoE
provisioned for it on the Storage subsystem. because everything above the transport layer
Zoning also ensures that unauthorized servers remains intact. By encapsulating native Fibre
don’t get access to Storage devices. FCoE based Channel in FCoE, the transparent interoperability
switches will continue to offer this zoning function of existing SAN networks will be possible.
ensuring that storage allocation and security
mechanisms are unaffected. Encapsulation

In addition to zoning, another integral element in The encapsulation of the Fibre Channel frame
SANs is the Name Server Database. This database occurs through the mapping of FC onto Ethernet.
covers a list of devices currently connected to Fibre Channel and traditional networks have stacks
the network. It also includes information about of layers where each layer in the stack represents
network location and status of the connections. a set of functionality. The Fibre Channel stack
FCoE based SANs will continue to offer the Name consists of five layers, FC-0 through FC-4. Ethernet
Server Database function. is typically considered a set of protocols in the
seven-layer OSI stack that define the physical and
Beyond the basic SAN provisioning functions data link layers. FCoE provides the capability to
mentioned above, Storage Administrators carry the FC-2 layer over the Ethernet layer. See
also deliver services such as Disaster Recovery, figure 2 below. This allows Ethernet to transmit
Backup and Recovery. Disaster Recovery solutions the upper Fibre Channel layers FC-3 and FC-4 over
typically used a protocol called FCIP. FCIP remains the IEEE 802.3 Ethernet layers. .
unchanged with FCoE. Data that needs to be sent
to a remote site will hop on to a FCIP link via a It is this FCoE mapping which allows FC traffic to
gateway and similarly enter pass over an Ethernet infrastructure.
a FCoE network via a FCIP
gateway. File Backups to
Tape Libraries and Recovery
from these devices will
continue to use existing
backup and recovery tools.
What about companies
who have an existing Fibre
Channel infrastructure?
Does FCoE mean that
they need to replace their Figure 2 – FCoE mapping
existing SANs? Absolutely
not. FCoE is designed to maintain the Fibre
Channel model. A storage administrator logged
into a server connected through an FCoE network
would see Fibre Channel initiators just as if that
same server were connected on a SAN. The same
management tools that customers use to manage
and maintain their SANs today can be used in an
FCoE environment.

www.fibrechannel.org © 2007 Fibre Channel Industry Association


Fibre Channel over Ethernet in the Data Center: An Introduction

Frame Format de-encapsulated and handed off seamlessly. This


How does FC traffic get passed over Ethernet? capability enables FCoE to integrate with existing
FCoE encapsulates a Fibre Channel frame within FC SANs without the need of a gateway.
an Ethernet frame. Figure 3 represents the frame Frame size is also a factor in FCoE. A typical Fibre
format as agreed to by the INCITS T11.3 standards Channel data frame has a 2112 byte payload,
body. a header and FCS. A classical Ethernet frame
is typically 1.5 KB or less. To maintain good
performance, FCoE must utilize jumbo frames
(or the 2.5 KB “baby jumbo”) to prevent a Fibre
Channel frame from being split into two Ethernet
frames.
Lossless Ethernet
One of the challenges with passing Fibre Channel
frames over Ethernet is that FC provides a lossless
transport. Congestion in a lossless network is
an issue that must be managed. Traditional
Fibre Channel manages congestion through the
use of a link-level, credit based flow control that
guarantees no loss of a frame under normal
conditions. Typical Ethernet, coupled with TCP/
Figure 3- The FCoE frame format IP, uses a packet drop flow control mechanism.
Packet drop flow control is not lossless making
The first 48-bits in the frame are used to specify it unacceptable for use with storage traffic.
the Destination MAC address and the next 48-bits Fortunately, classical Ethernet has a PAUSE
specify the Source MAC Addresses. The 32-bit capability so that a busy receive port can send a
IEEE 802.1Q Tag provides the same function as it control frame to the transmit port requesting a
does for virtual LANs, allowing multiple virtual pause in transmission. Use of this optional IEEE
networks across a single physical infrastructure. 802.3x Ethernet feature enables Fibre Channel
FCoE has its own Ether type as designated by the traffic to pass on an Ethernet network in a lossless
next 16 bits, followed by the 4-bit version field. fabric. See Figure 4 below.
The next 100-bits are reserved and are followed
by the 8-bit Start of Frame and then the actual
FC frame. The 8-bit End-of Frame delimiter is
followed by 24 reserved bits. The frame ends with
the final 32-bits dedicated to the FCS function that
provides error detection for the Ethernet frame.
The encapsulated Fibre Channel frame consists of
the original 24 byte FC header and the data being
transported (including the Fibre Channel CRC). The
CRC is a cyclical redundancy check used for error
detection. The FC header is maintained so that
when a traditional FC Storage Area Network is
connected to an FCoE capable switch the frame is
Figure 4 – Lossless Fibre Channel and lossless Ethernet

www.fibrechannel.org © 2007 Fibre Channel Industry Association


Fibre Channel over Ethernet in the Data Center: An Introduction

There is also a new Ethernet enhancement being Bridge where it is distributed to the appropriate
developed that allows PAUSE functionality to Ethernet port on the bottom of the diagram.
be enabled for each user-priority supported
in Ethernet. While PAUSE provides the basic The reverse is also true; FCoE traffic will flow from
functionality to make Ethernet lossless, a new the red ports, through the Ethernet Bridge until it
proposal for Priority Flow Control will provide is de-encapsulated inside of the FCoE Entity, where
significant benefit for both FC and Ethernet traffic. it passes through the FC switch to the appropriate
Priority Flow Control is discussed on page 7. FC port.

Example of an FCoE switch This is just one example of how a switch can be
FCoE enabled. Different vendors may choose to
Figure 5 shows a functional Diagram of a FCoE implement FCoE in different ways.
capable switch as depicted by the larger, yellow
field. Inside this device is both an Ethernet FC and FCoE address mapping
switch function as well as a Fibre Channel A third component of FCoE mentioned above is
switch function. Note the Fibre Channel over replacing of a Fibre Channel link MAC addresses
Ethernet entity within the FC Switch. This entity in a lossless Ethernet fabric. Traditional Fibre
encapsulates FC frames into FCoE Frames and de- Channel fabric switches maintain forwarding
encapsulates FCoE frames back to FC frames. FC tables FC_IDs. FC switches use these forwarding
traffic flows into one end of the FCoE entity and tables to select the best link available for a frame
FCoE traffic flows out the other end. so the frame reaches its destination port. Fibre
The FCoE Entity has an Ethernet MAC address that Channel links are typically point-to-point and do
will be used as a source or destination address not need an address at the link layer. An Ethernet
when FCoE frames are transmitted through an network is different because it does not form an
Ethernet fabric (Further details provided below). end-point to end-point connection in the manner
of FC. This requires FCoE to rely on Ethernet MAC
In figure 5, an FC frame enters from the green FC addresses to direct a frame to its correct Ethernet
ports to the right and into the FC switch where the destination (the first two fields of the FCoE frame
entity encapsulates the Fibre Channel frame into a are MAC addresses, as outlined previously in figure
FCoE frame. The FCoE frame is sent to an Ethernet 3).

Figure 5 – Functional diagram of an FCoE switch.

www.fibrechannel.org © 2007 Fibre Channel Industry Association


Fibre Channel over Ethernet in the Data Center: An Introduction

Figure 6 – FC and FCoE address mapping


Figure 6 illustrates how traditional FC addresses While the original FC source and destination ID’s
align with MAC addresses in an FCoE network. are maintained in the encapsulated FC frame,
On the left is a storage array attached to a Fibre the FCoE entity will populate a new destination
Channel switch labeled FC Domain 7. This storage and source MAC address located in the Ethernet
array is in a traditional SAN and stores information header. As mentioned above, the entity has its
for an Engineering host on an FCoE enabled Fabric. own MAC address. In this case, the destination
The Engineering host has both a FC_ID 1.1.1 and a is MAC address B (the MAC address of the FCoE
FC MAC address. entity in the receiving switch) and source MAC
address A (the MAC address of the FCoE entity in
Figure r Step by Step Analyses the transmitting switch).
1. The Fibre Channel N_port on the storage
array sends out the FC frame, which includes the 4. When the FCoE frame arrives at the FCoE
Destination FC_ID (D_ID = 1.1.1) and the Source entity with MAC address B, the frame is de-
FC_ID (S_ID = 7.1.1) in the header. For simplicity, encapsulated and the switch determines that
only the header information is displayed in the the FC frame destination is within its domain
diagram’s frame graphic (Domain 1). The FC frame is re-encapsulated
with the new destination MAC address C (which
2. The Fibre Channel switch with Domain ID 7 corresponds to the FC D_ID 1.1.1) and the
receives the frame. Since the destination ID (D_ new source MAC address B. Then the frame is
ID) in not in this FC domain (7), the switch looks transmitted out the appropriate port to the FCoE
up the destination domain ID in its forwarding host with MAC address C.
table and transmits the frame on the port
associated with the shortest path, as determined 5. When the frame is received by the FCoE host
by the Fabric Shortest Path First algorithm (FSPF). bus adapter with MAC address C, the FCoE frame
3. The switch with the FC Domain ID 3 receives is de-encapsulated and the FC frame accepted by
the frame and determines that the destination the Engineering host with FC_ID 1.1.1.
ID (D_ID) is not in Domain 3 and repeats the This example demonstrates how FCoE addressing
lookup process in step 2. However, in this case maps to traditional FC addressing. Topologies will
the FC frame will be transmitted across an vary depending on a customer’s implementation
FCoE enabled Ethernet fabric. This requires the requirements and the FCoE capable products
frame to be encapsulated by an FCoE entity in deployed. While the topologies will change, the
the switch and then transmitted on the port addressing essentials will remain constant.
associated with the shortest path.

www.fibrechannel.org © 2007 Fibre Channel Industry Association


Fibre Channel over Ethernet in the Data Center: An Introduction

Example of an FCoE topology Ethernet technologies that enhance FCoE


FCoE can be deployed in a variety of topologies, Previous examples highlighted the ability for
depending on the business needs of a company FCoE to transport FC traffic over an Ethernet
and the products chosen. Figure 7 depicts a simple infrastructure. In these examples FCoE could be
FCoE topology where I/O consolidation is achieved viewed simply as a method for unifying I/O over
at the access layer of the network. FCoE is used in Ethernet, an alternative to a pure Fibre Channel
this scenario to consolidate traffic from a server storage area network. However, there are ongoing
to FCoE enabled switches. The FCoE switches technological developments that will make FCoE
pass Fibre Channel traffic to attached SANs and invaluable in tomorrow’s data center.
Ethernet traffic to an attached Ethernet network.
This deployment model targets the installed base Examining bandwidth requirements is a good
of LAN and SAN environments in the datacenter place to start. Many customers use 1/2/4 Gbps
providing a phased approach to I/O consolidation FC connections in their datacenters today. Passing
adoption. a 2 or 4 Gbps FC link over a one Gigabit Ethernet
connection doesn’t make a lot of sense. However,
on a 10 Gigabit Ethernet link customers have the
bandwidth capacity to easily pass 4 Gbps of FC.
Moreover, there is plenty of bandwidth remaining
in the 10 GE link to allow Ethernet traffic to pass
simultaneously. Networks also could pass 8 Gbps
FC over a FCoE link.
Another extension to Ethernet being developed
that will support FCoE is the IEEE 802.1p Class
Based Flow Control (aka CBFC, formerly called Per
Priority Pause/ Priority Flow Control) functionality.
The 3-bit field in IEEE 802.1p allows the creation
of logical lanes on a single, physical fabric. A
logical lane can be lossless when PAUSE can be
implemented on a per lane basis. The per flow
PAUSE function remains optional so customers
could implement one or two lossless “lanes” for
Figure 7 – A simple FCoE Topology FC while allowing others to pass regular Ethernet
traffic.
There are many FCoE deployment models that
will be utilized by customers. Another example Flow Control is another Ethernet enhancement
would be to extend the reach of FCoE into that also can improve FCoE’s value in customer
the aggregation and core layers of a network. data centers. Fibre Channel uses a link level flow
Yet another example will be an end-to-end control with port based buffer-to-buffer credits
deployment of an entirely FCoE network. All to control congestion in the link. A lossless
of these topologies are provided for by the Ethernet fabric using the per flow PAUSE described
proposed T11 FC-BB-5 standard which mandates above can prevent the frame drop today, but an
connectivity to existing infrastructures. additional option, Congestion Notification, can be
utilized to enhance the network for FCoE traffic.

www.fibrechannel.org © 2007 Fibre Channel Industry Association


Fibre Channel over Ethernet in the Data Center: An Introduction

IEEE 802.1au Congestion Notification (CN) is


a proposal which provides options for acting
upon congestion in the network. CN will provide
additional flow control capabilities for a network
implementing the CBFC function. One option
is a CN where congestion points in the network
notify a rate limiter when congestion occurs, so
that the traffic can be throttled back. Another,
more complex CN option is a 3-point flow control
scheme using in-band and out of band signaling
to provide feedback to rate limiters. While a
detailed review of CN is outside the scope of this
document, readers interested in the proposal
should refer to:
http://www.ieee802.org/1/pages/802.1au.html.
As FCoE continues to evolve further into the
network, technologies such as Congestion
Notification will play an increasingly important
role.

Conclusion
Fibre Channel over Ethernet will allow companies’
data centers to optimize increases in server
virtualization for the next several years. FCoE
enables I/O consolidation and offers these
enhancements to the data-center:
• Fewer Network Interface Cards per server
• Reduction in server power budget and reduced
cooling requirements
• Significant reduction in the amount of cables
to manage
• Seamless connectivity with existing Storage
Area Networks and Ethernet Networks –
customers can continue to leverage their existing
FC tools, security model and infrastructure
FCoE positions Fibre Channel as the storage
networking protocol of choice and extends the
reach of Fibre Channel throughout the data center
to all servers.
Readers interested in learning more about FCoE
can refer to the following sources:
www.t11.org/FCoE
www.fibrechannel.org

www.fibrechannel.org © 2007 Fibre Channel Industry Association

You might also like