Data Center Bridging Standards / Converged Enhanced Ethernet
Enhancements required for a successful Unified Fabric implementation, where network and storage traffic are carried over the same Ethernet links, are defined under the Data Center Bridging standards defined by the IEEE.
These enhancements make sure that Unified Ethernet fabric provides lossless, no-drop, in-order delivery of packets end-to-end.
This is also called Converged Enhanced Ethernet (CEE).
These enhancements include:
- Priority Flow Control (PFC)
- Enhanced Transmission Selection (ETS)
- Quantized Congestion Notification (QCN)
- Data Center Bridging Exchange (DCBX)
Priority Flow Control
PFC provides lossless Ethernet transport services for FCoE traffic via link-layer signalling that keeps track of buffers at the remote end.
PFC enables the Ethernet flow control mechanism to operate on the per-802.1p Class of Service level, thus selectively pausing at Layer 2 only traffic requiring guaranteed delivery, such as FCoE. All other traffic is subject to regular forwarding and queuing rules.
With PFC, PAUSE frames are signalled back to the previous hop DCB-capable Ethernet switch for the negotiated FCoE Class of Service (CoS) value (Class of Service value of 3 by default). PAUSE frames are transmitted when the receive buffer allocated to a no-drop queue on the current DCB-capable switch is exhausted.
This process is known as a back pressure, and it can be triggered hop-by-hop from the point of congestion toward the source of the transmission (FCoE initiator or target).
Enhanced Transmission Selection
ETS creates priority groups by allowing differentiated treatment for traffic belonging to the same Priority Flow Control class of service based on bandwidth allocation, low latency, or best effort.
Bandwidth allocations made by ETS are not hard allocations, meaning that if a particular class of service isn’t using its full allocation of bandwidth, its spare allocation is available for other classes to borrow.
ETS leverages advanced traffic scheduling features by providing guaranteed minimum bandwidth to the FCoE and high-performance computing traffic.
The default behaviour of Cisco switches supporting DCB standards, such as Cisco Nexus 5000 Series switches, is to assign 50% of the interface bandwidth to FCoE no-drop class.
Data Center Bridging Exchange
Data Center Bridging Exchange (DCBX) is a negotiation and exchange protocol between FCoE
initiator/target and a DCB-capable switch or between two DCB-capable switches. It is an extension of the Link Layer Discovery Protocol (LLDP) defined under the IEEE 802.3AB standard
DCBX performs the discovery of adjacent DCB-capable switches, detects configuration mismatches, and negotiates DCB link parameters.
PFC and ETS are both examples of link-layer features that can be negotiated and configured via DCBX.
Quantized Congestion Notification (QCN)
Quantized Congestion Notification mechanism, defined in the IEEE 802.1Qau standard, takes a different approach of applying back-pressure from the point of congestion directly toward the source of transmission without involving intermediate switches.
Even though Layer 2 operation implies bridging, the MAC address swapping behaviour occurring at FCoE Fibre Channel Forwarders makes its operation resemble routing more than bridging.
Because those FCoE frames no longer contain the original MAC address of the initiator, it is impossible for the intermediate FCoE Fibre Channel Forwarders to generate QCN messages directly to the source of transmission to throttle down its FCoE transmission, unless all FCoE Fibre Channel Forwarders along the path from the point of congestion toward the source of transmission participate in the QCN process.
Participation of all FCoE Fibre Channel Forwarders in the QCN process brings little value beyond Priority Flow Control and as such, Cisco Nexus 5000 Series DCB-capable switches do not implement the QCN feature.
Fibre Channel over Ethernet (FCoE)
Fibre Channel frames are statelessly encapsulated within layer 2 Ethernet frames and transported over an IEEE 802.3 wired Ethernet network.
Ethernet is just a new physical transport medium, replacing the lower layers of the FC stack, FC-0 and FC-1.
The Storage Networking Industry Association (SNIA) defines baby jumbo frames as frames that are 2.5 KB in length and states that FCoE encapsulation requires at least baby jumbo (2.5KB) Ethernet frames.
The total size of an Ethernet frame carrying a Fibre Channel payload can be up to 2180 bytes.
This encapsulation of FCoE frames is a hop-by-hop process, meaning that each and every FCoE switch in the network has to strip off the FCoE encapsulation on receipt of an FCoE frame and then re-encapsulate the frame before forwarding it on to the next hop. The FCoE encapsulation process leaves the FC frame intact.
FCoE Logical Endpoint
The FCoE Logical Endpoint (FCoE_LEP) is responsible for the encapsulation and de-encapsulation of the FC traffic into the Ethernet frames. FCoE itself takes FC layers 2-4 and maps them to Ethernet layers 1-2, this replaces the FC-0 Physical layer, and FC-1 Encoding Layer. This mapping between Ethernet and Fibre Channel is done through a Logical End-Point (LEP) which can be thought of as a translator between the two protocols.
An FCoE End Node (ENode) is an FC node that is associated with one or more Enhanced Ethernet MAC, one or more FCoE Logical End Point and one or more VN_Ports.
FCoE ENodes could be either initiators in the form of servers or targets in the form of storage arrays. Each one presents a VN_Port port type connected to the VF_Port port type on the first hop upstream FCoE switch.
Converged Network Adapters
Converged Network Adapter (CNA) is an FCoE ENode. CNA is usually a PCI express (PCIe) type adapter, which contains the functions of a traditional Host Bus Adapter (HBA) and a network interface card (NIC) all in one.
From the operating system perspective, CNAs are represented as two separate devices; as such, they still require two sets of drivers.
Each VN_Port has a unique ENode MAC address, which is dynamically assigned to it by the FCoE Fibre Channel Forwarder on the upstream FCoE switch during the fabric login process. This MAC is also known as the fabric-provided MAC address (FPMA).
FPMA is 48 bits in length and consists of two concatenated 24-bit fields. The first field is
called FCoE MAC address prefix, or FC-MAP, and it is defined by the FC-BB-5 standard as a range
of 256 prefixes. The second field is called FC_ID, and it is basically a Fibre Channel ID assigned to
the CNA during the fabric login process.
An ENode also has a globally unique burnt-in MAC address.
Fibre Channel Forwarder
Although FCoE frames are transported over Converged Enhanced Ethernet networks, the forwarding of FCoE frames is not the responsibility of Ethernet forwarding. Instead, FCoE forwarding is the responsibility of a function in the FCoE switch known as the FCoE Forwarder (FCF).
FCoE Fibre Channel Forwarder (FCF) is a logical Fibre Channel switch entity residing in the FCoE-capable Ethernet switch. The FCF forwards FCoE frames based on the FC Destination_ID (D_ID) field of the encapsulated FC frame, as well as a routing table created by the Fabric Shortest Path First (FSPF) protocol.
When the FCoE endpoint sends FCoE frames to the network, these frames include the source MAC address of the endpoint and the destination MAC address of the FCoE Fibre Channel Forwarder.
When these frames reach the FCoE FCF, frames are de-encapsulated, Ethernet headers are removed, and a Fibre Channel forwarding decision is taken on the exposed Fibre Channel frame. Subsequently, the Fibre Channel frames are re-encapsulated with Ethernet headers, while source and destination MAC addresses assigned are the MAC addresses of the current and next-hop FCoE FCF or the frame target, respectively.
FCoE Fibre Channel Forwarder is also responsible for the host fabric login, as well as the encapsulation and de-encapsulation of the FCoE traffic. One or more FCoE_LEPs are used to attach the FCoE Fibre Channel Forwarder internally to the Ethernet switch.
FCoE Port Types
- The VN_Port is the virtual Node interface instantiated by the ENode. The VN_Port mimics the same function as N_ Port in the native Fibre Channel.
- The VF_Port is the Virtual Fabric port instantiated by the FCF. The VF_Port mimics the same function as F_Port in the native Fibre Channel.
- The VE_Port is the Virtual Ethernet Port instantiated by the FCF. The VE_Port mimics the same function as E/TE port in the native Fibre Channel.
FCoE Initialization Protocol
FCoE Initialization Protocol (FIP) is the FCoE control plane protocol responsible for establishing and maintaining Fibre Channel virtual links between pairs of adjacent FCoE devices (FCoE ENodes or FCoE Fibre Channel Forwarders), potentially across multiple Ethernet segments.
Establishment of the virtual link carries three main steps: discovering the FCoE VLAN, discovering the MAC address of an upstream FCoE Fibre Channel Forwarder, and ultimately performing fabric login/discovery operation (FLOGI or FDISC).
After the virtual link has been established, an exchange of Fibre Channel payload encapsulated with the Ethernet headers can begin.
The FCoE ENode discovers the VLAN to which it needs to send the FCoE traffic by sending an FIP VLAN discovery request to the All-FCF-MAC multicast MAC address. All FCoE Fibre Channel Forwarders listen to the All-FCF-MAC MAC address on the native VLAN.
The FCoE ENode can also be manually configured for the FCoE VLAN it should use, in which case FIP VLAN discovery functionality is not invoked. Cisco implements the default FCoE VLAN over 1002.
After FCoE VLAN has been discovered, FIP performs the function of discovering the upstream FCoE Fibre Channel Forwarder, which can accept and process fabric login requests. FCoE Fibre Channel Forwarder discovery messages are sent to the All-ENode-MAC multicast MAC address over the FCoE VLAN discovered earlier by the FIP protocol.
Discovery messages are also used by the FCoE Fibre Channel Forwarder to inform any potential FCoE ENode attached through the FCoE VLAN that its VF_Ports are available for virtual link establishment with the ENode’s VN_Ports.
The FC-BB-5 standard enables FCoE ENodes to send out advertisements to the All-FCF-MAC multicast MAC address to solicit unicast discovery advertisement back from FCoE Fibre Channel Forwarder to the requesting ENode.
Following the discovery of FCoE VLAN and the FCoE Fibre Channel Forwarder, FIP performs
fabric login or discovery by sending FLOGI or FDISC unicast frames from VN_Port to VF_Port.
This is the phase when the FCoE ENode MAC address used for FCoE traffic is assigned and virtual link establishment is complete.
FCoE packets must be exchanged in a VLAN. The FCoE traffic belonging to different VSANs must remain separated by different VLANs on the Ethernet plane. This choice simplifies the implementation, since it removes the necessity to include both a VLAN and a VSAN header in each FCoE packet.
For this reason, the Cisco Nexus 5000 Series software introduced the concept of a mapping table between FCoE VLANs and Fibre Channel VSANs. For each Fibre Channel VSAN used in the Fibre Channel fabric, the administrator associates one and only one unique FCoE VLAN; all FCoE packets tagged with that VLAN ID are then assumed for Fibre Channel control plane and forwarding purposes to belong to the corresponding Fibre Channel VSAN.
The Cisco Nexus 5000 Series implementation also expects all FCoE VLANs to be used exclusively for FCoE traffic and never shared by other traditional Ethernet payloads (for example, IP traffic).
Data Storage Networking, Nigel Poulton, Wiley, 2014
Cisco CCNA Data Center DCICT 640-916 Official Certification Guide, Navaid Shamsee, et al, Cisco Press, 2015