Table of Contents
Table of Contents
Fibre Channel, or FC, is a high-speed network technology (commonly running at 1, 2, 4, 8, 16, and 32 gigabit per second rates) primarily used to connect computer data storage to servers. Fibre Channel is primarily used in storage area networks (SAN) in enterprise storage. Fibre Channel networks are known as a fabric because they operate in unison as one big switch. Fibre Channel mainly runs on optical fiber cables within and between data centers.
FC Protocol Stack
FCP is a layered protocol
FC-0 defines the physical transmission medium (cable, plug) and specifies which physical signals are used to transmit the bit s 0 and 1.
FC-1 defines how data is encoded before it is transmitted via a Fibre Channel cable FC-1 also describes certain transmission words (ordered sets) that are required for the administration of a Fibre Channel connection (link control protocol).
The encoding process ensures that sufficient clock information is present in the serial data stream. This information allows the receiver to synchronize to the embedded clock information and successfully recover the data at the required error rate. (e.g. 8b/10b encoding)
Every 8 bits of data are encoded into a 10 bit Transmission Character. A Transmission Word is composed of four contiguous Transmission Characters. There are two categories of transmission words: data words and ordered sets.
Ordered Sets are Transmission Words that allow bit or word synchronization and word boundary alignment. The three types of Ordered Sets are:
- Frame Delimiters
- (SOF) Start-of-Frame
- (EOF) End-of-Frame
- Primitive Signals
- (R_RDY) Receiver Ready
- Primitive Sequence
- (OLS) Off-line
- (NOS) Not Operational
- (LR) Link Reset
- (LRR) Link Reset Response
FC-2 provides the following functions:
- Frame format definition
- Flow Control (classes of service)
- Sequence disassembly and reassembly
- Exchange management
- Address assignment
- Alias address definition
A Fibre Channel network transmits control frames and data frames. Control frames contain no useful data; they signal events such as the successful delivery of a data frame. Data frames transmit up to 2,112 bytes of useful data that must be divisible by 4 (Remember Transmission Words)..
Start of Frame: All FC frames are prefaced with a start of frame (SOF) ordered set that
acts as a form of frame delimiter. The SOF is a single 4-byte word indicating the class of
service (CoS) and at what point this particular frame figures in a set.
Header: Directly following the SOF is a 24-byte frame header. This contains the source
and destination N_Port IDs, referred to as source ID (SID) and destination ID (DID). The
header also defines whether the frame contains user data (data frame) or control data (link
control frame) and indicates where a frame is featured in an exchange or sequence.
Payload: Following the header comes the payload. This can be up to 2,112 bytes long and
must be padded if not divisible by 4.
Cyclic Redundancy Check: Frame integrity is possible via a cyclic redundancy check
End of Frame: Last but not least comes the end of frame (EOF) ordered set. This informs
the receiving node that the frame is ended, and the specifics of the EOF are determined by
the class of service of the frame.
Fibre Channel Flow Control
Flow control ensures that the transmitter sends data only at a speed that the receiver can receive it. Fibre Channel uses credit model. Each credit represents the capacity of the receiver to receive a Fibre Channel frame.
FC-2 defines two mechanisms for flow control:
- End-to-End flow control
- Buffer to buffer (B2B) flow control
End-to-End flow control
In end-to-end flow control, two end devices such as an initiator and a target negotiate the end-to-end credit before the exchange of data. ACK frames are used to replenish credits. The end-to-end flow control is realized on the HBA cards of the end devices.
Buffer to buffer (B2B) flow control
Buffer to buffer (B2B) control takes place at each physical connection. Two communicating ports negotiating the buffer-to-buffer credits achieve this. Buffer to buffer control flow uses a special receiver ready frame (R_RDY) to indicate that the recipient is ready to receive more frames.
Fibre Channel Service Classes
Connectionless but utilizes acknowledgment frames. Class F is the defacto COS used for switch-to-switch communications. It operates only across E_Ports and ISLs. It is used for delivery of control data such as state-change notifications, name server updates, routing info, zoning, build fabric requests, and reconfigure fabric requests. Class F uses the B2B flow control mechanism.
Sequence and Exchange Management
A Sequence is formed by a set of one or more related Frames transmitted unidirectionally from one N_Port to another. Each Frame within a sequence is uniquely numbered with a Sequence Count. Error recovery, controlled by an upper protocol layer is usually performed at Sequence boundaries meaning that if a frame is dropped or lost, the entire sequence must be re-sent.
An Exchange is composed of one or more non concurrent sequences for a single operation. The Exchanges may be unidirectional or bidirectional between two N_Ports. Within a single Exchange, only one sequence may be active at any one time, but Sequences of different Exchanges may be concurrently active.
Fibre Channel Addressing
Fibre Channel uses World Wide Names (WWN) and Fibre Channel IDs (FCID) for addressing.
World Wide Names
Each node port has its own 64-bit (8-byte) worldwide unique number that stays with its device for its entire life. This 64-bit name is known as a worldwide name (WWN).
The two types of WWNs are node WWNs (nWWN) and port WWNs (pWWN):
The nWWNs uniquely identify devices. Every HBA, array controller, switch, gateway, and Fibre Channel disk drive has a single unique nWWN.
The pWWNs uniquely identify each port in a device. A dual-ported HBA has three WWNs. one nWWN and one pWWN for each port.
Ports must be uniquely identifiable because each port participates in a unique data path. The nWWNs are required because the node itself must sometimes be uniquely identified. For example, path failover and multiplexing software can detect redundant paths to a device by observing that the same nWWN is associated with multiple pWWNs.
FC ID is a 24-bit dynamically assigned address. It is dynamically assigned by the fabric when an FC ID logs in to the fabric. FC IDs are used for frame switching and delivery on FC networks.
The format and structure of FC IDs is as follows:
Domain ID: The first 8 bits of the address are known as the domain ID. It is the domain ID of the switch the port belongs to. Each switch receives a unique domain ID.
Although the domain ID is an 8-bit field, only 239 domains are available to the fabric: Domains 01 through EF are available.
Domains 00 and F0 through FF are reserved for use by switch services.
Each fabric has one, and only one, principal switch. The principal switch manages the distribution of domain IDs within the fabric. They are also the authoritative source of time in a fabric. Each switch must have a unique domain ID, so there can be no more than 239 switches in a fabric.
Area ID: The area ID is used to identify groups of ports within a domain. Areas can be used to group ports within a switch. Areas are also used to uniquely identify fabric-attached arbitrated loops. Each fabric-attached loop receives a unique area ID.
Port ID: The last 8 bits of the address are also vendor specific and usually translate to the
port number of the port in the switch. The value can be anything from 0–255, and switches
with more than 256 ports use a combination of area and port to uniquely identify the port.
pWWNs are long and mean nothing to the average human being. Aliases allow us to give friendly names to pWWNs on the SAN.
There are five different protocols for Fibre Channel not to be confused with upper level protocols”
- Primitive Sequence Protocols
Tease are based on Primitive Sequences and specified for link failure.
- Fabric Login protocol:
The interchanging of Service Parameters of an N_Port with the fabric.
- N_Port Login protocol:
Before performing data transfer, the N_Port interchanges its Service Parameters with another N_Port.
- Data transfer protocol
These protocols describe the methods of transferring Upper Layer Protocol (ULP) data using the Flow control management of Fibre Channel.
- N_Port Logout Protocol
This is performed when an N_Port requests removal of its Service Parameters from the other N_Port. This may be used to free up resources at the connected N_Port.
There are a set of services available to all devices participating in a fabric. They are known as fabric services, and include:
- Management services
- Time services
- Name services
- Login services
- Registered State Change Notification (RSCN
This is an in-band fabric service which allows data to be passed from devices to management platforms. This will include such information as the topology of the SAN. A critical feature of this service is that it allows management software access to the SNS bypassing any potential block caused by zoning. This means that a management suite can have a view of the entire SAN.
This is defined, but has not yet been implemented at the time of writing.
Fabric switches and directors implement a concept known as the name server, or Simple Name Server, or SNS. All switches and directors in the fabric keep the name server updated, and are therefore aware of all other devices in the name server. After a node has successfully logged into the fabric, it registers itself and passes on critical information such as class of service parameters, its WWN, and the Upper Layer Protocols which it can support.
In order to do a fabric login, a node communicates with the login server.
Registered State Change Notification
This service, Registered State Change Notification (RSCN), is critical as it propagates information about a change in state of one node to all other nodes in the fabric. This means that in the event of, for example, a node being shutdown, the other nodes on the SAN will be informed and can take the necessary steps to stop communicating with it
FC-3 provides the Fibre Channel Generic Services (FC-GS) that is required for fabric management. Specifications exist here but are rarely implemented.
FC-4 provides protocol mapping to identify the ULP that is encapsulated in to a protocol data unit (PDU) for delivery to the FC-2 layer.
FCP is the FC-4 mapping of SCSI -3 on t o FC.
Fibre Connection (FICON) is a further important application protocol. FICON maps the ESCON protocol (Enterprise System Connection) used in the world of mainframes onto Fibre Channel networks.
Note that FCP does not define its own header. Instead, fields with in the FC-2 header are used by FCP.
Fibre Channel SAN Components
Hosts and servers connect to the SAN via one or more Fibre Channel host bus adapters
(HBAs) that are installed on the PCIe bus of the host. HBAs offload all FCP-related overhead from the host CPU and memory.
Also, most HBAs come with a BIOS, meaning that the servers they are installed in can boot from SAN.
FC Storage Arrays
In an FC SAN, a storage array is an end point/end device with one or more node ports (N_Ports). These node ports are configured in Target mode—and therefore act as SCSI targets—accepting SCSI commands from SCSI initiators such as server-based HBAs
Switches and directors provide connectivity between end devices such as hosts and storage. They operate at layers FC-0, FC-1, and FC-2.
A fabric is a collection of connected FC switches that have a common set of services. For
example, they share a common name server, common zoning database, common FSPS routing table, and so on. Each fabric is viewed and managed as a single logical entity.
Common SAN Topologies
Point-to-point, referred to technically as FC-P2P, is a direct connection from a host HBA
port to a storage array port.
Fibre Channel arbitrated loop, referred to as FC-AL, allows devices to be connected in a
loop topology, usually via an FC hub. However, a hub is not required, as you can daisy chain servers together in a loop configuration where the transmit port of one server is connected to the receive ports of the adjacent server.
A Fibre Channel fabric is made up of one or more FC switches. FC switches operate up
to FC-2 layer, and each switch supports and assists in providing a rich set of fabric services
such as the FC name server, the zoning database, time synchronization service.
SAN Topology Considerations
It is common practice in SAN environments to build two separate, redundant physical fabrics (Fabric A and Fabric B) in case a single physical fabric fails. When designing for large networks, most environments will fall into two types of topologies within a physical fabric:
- Two-tier: Core-edge design
- Three-tier: Edge-core-edge design
Within the two-tier design, servers connect to the edge switches, and storage devices connect to one or more core switches.
Three-tier design still uses a set of edge switches for server connectivity, but adds another set of edge switches for storage devices. Both sets of edge switches connect into a core switch via ISLs.
Fibre Channel—Standard Port Types
- Node Port (N port): Node ports used to connect devices to switched fabric.
- Expansion port (E Port): This port connects to another E Port to create an inters witch link (ISL) between two switches.
- Fabric port (F Port): This port connects to a peripheral device (such as a host or disk) that operates as an N Port.
- Trunking Node Port (TN port): Trunking Node port carries several VSANs.
- Trunking expansion port (TE Port): This port connects to another TE Port to create an extended ISL (EISL) between two switches. TE Ports are specific to Cisco MDS 9000 Series switches and support virtual SAN (VSAN) trunking, transport quality of service (QoS) parameters, and the Fibre Channel Traceroute (fctrace) feature.
- Trunking fabric port (TF Port): This interface connects to another trunking node port (TN Port) or trunking node-proxy port (TNP Port) to create a link between a core switch and an NPV switch or a host bus adapter (HBA) to carry tagged frames. TF Ports are specific to Cisco MDS 9000 Series switches and expand the functionality of F Ports to support VSAN trunking. In TF Port mode, all frames are transmitted in the EISL frame format, which contains VSAN information.
- Node-proxy port (NP Port): An NP Port is a port on a device that is in N-Port Virtualization (NPV) mode and connects to the core switch via an F Port. NP Ports function like node ports (N Ports) but in addition to providing N Port operations, they also function as proxies for multiple physical N Ports.
- TNP Port: In TNP Port mode, an interface functions as a trunking expansion port. This interface connects to a TF Port to create a link to a core N Port ID Virtualization (NPIV) switch from an NPV switch to carry tagged frames.
- Switched Port Analyzer (SPAN) destination port (SD Port): In SD Port mode, an interface functions as a SPAN. The SPAN feature is specific to Cisco MDS 9000 Series switches. An SD Port monitors network traffic that passes through a Fibre Channel interface.
- SPAN tunnel port (ST Port): In ST Port mode, an interface functions as an entry-point port in the source switch for the Remote SPAN (RSPAN) Fibre Channel tunnel. ST Port mode and the RSPAN feature are specific to Cisco MDS 9000 Series switches. When a port is configured as an ST Port, it cannot be attached to any device and therefore cannot be used for normal Fibre Channel traffic.
- Fx Port: An interface that is configured as an Fx Port can operate in either F or FL Port mode. Fx Port mode is determined during interface initialization, depending on the attached N or NL Port.
- Bridge port (B Port): Whereas E Ports typically interconnect Fibre Channel switches, some SAN extender devices implement a B Port model to connect geographically dispersed fabrics.
Fibre Channel Login Process
Fabric Login (FLOGI)
An N_Port will send a FLOGI frame that contains its Node Name, its N_Port Name, and service parameters to a well-known address of 0xFFFFFE to the switch. The switch accepts the login and returns an accept (ACC) frame to the sender. If some of the service parameters requested by the N_Port are not supported, the switch will set the appropriate bits in the ACC frame to indicate this.
The FLOGI process performs a bunch of important functions:
- Assigning devices their all-important 24-bit N_Port IDs
- Specifying the class of service to be used
- Establishing a device’s initial credit stock
Port Login (PLOGI)
After a successful FLOGI process, the device then must performs a port login (PLOGI) to the name server to register its capabilities. The device registers its 24-bit port address, 64-bit Port Name (pWWN), 64-bit Node Name (nWWN), class of service parameters, FC-4 protocols supported, and port type.
As part of the PLOGI, the device will ask the name server for a list of devices on the fabric. The initiator queries the name server with the WWN of the target device. The name server looks up and returns the current port address that is associated with the target WWN. Then a session between two N_Ports (devices) is established and session parameters are exchanged between the two N_Ports.
Process login is also known as PRLI. Process login is used to establishes communication between two FC-4 layer processes (SCSI), between two different N_Ports
The FC network is built on two separate networks (commonly called pat h A and path B), and each end node (host or storage) is connected to both network s. Multipathing establishes two or more SCSI communication connections between a host system and the storage it uses. If one of these communication connections fails, another SCSI communication connection is used in its place.
FSPF is the standard routing protocol used in Fibre Channel fabrics. FSPF automatically calculates the best pat h between any two devices in a fabric through dynamically computing routes, establishing the shortest and quickest path between any two devices. It also selects an alternative pat h in the event of failure of the primary path.
Zoning provides security within a single fabric, whether physical or logical, to restrict access between initiators and targets. There are two main methods of zoning; hard and soft.
Soft zoning restricts only the fabric name service, to show only an allowed subset of devices. In hard zoning the switch hardware inspects all traffic crossing the fabric and actively filters and drops frames that are disallowed according to the zoning configuration. Name server zoning is still in effect.
Multiple zones are grouped together in to zone sets, and it is the zone set that is applied to
the fabric. If you have configured a new zone, you will need to add it to the active zone set
in order for it to be applied to the fabric.
LUN masking is essentially a mapping table inside the front-end array controllers. LUN masking determines which LUNs are to be advertised through which storage array ports, and which host is allowed to own which LUNs.
An alternative method of LUN security is LUN mapping. If LUN masking is unavailable in the storage array, LUN mapping can be used, although both methods can be used concurrently.
LUN mapping is configured in the HBA of each host, to ensure that only one host at a time can access each LUN.
Virtual Storage-Area Network (VSAN)
VSANs are used to segment the physical fabric into multiple logical fabrics. Each VSAN is a fully functioning fabric, with its own dedicated ports, devices, and fabric services. These fabric services include the usual suspects of the name server, zoning table, alias table, and so on.
Inter-VSAN Routing (IVR)
Using IVR, you can access resources across VSANs without compromising other VSAN benefits. IVR works by creating zones and adding them to the IVR zone set. Zones in the IVR zone set effectively overlap VSANs, allowing devices that are in separate VSANs but exist in the IVR zone set to communicate with each other.
N_Port ID Virtualization
A node supporting NPIV registers multiple virtual N_Ports with the fabric. Registration of additional N_Ports on the fabric is done via FDISC operations. Basically, an N_Port can perform as many FDISC operations as it wishes, with each FDISC followed by a PLOGI for
the acquired N_Port ID.
A switch in NPV mode is invisible to the fabric. It doesn’t register for a domain ID and it doesn’t share the burden of assisting with fabric services. As far as the fabric is concerned, the switch is not a switch; it just looks like an N_Port running NPIV.
NPV switches on the edge effectively proxy all of the FLOGI and PLOGI operations from the end devices connected to them.
Data Storage Networking, Nigel Poulton, Wiley, 2014
Cisco CCNA Data Center DCICT 640-916 Official Certification Guide, Navaid Shamsee, et al, Cisco Press, 2015
Introduction to Storage Area Networks, Jon Tate, et al, IBM Redbooks, 2016