The current general feeling amongst lay IT people when it comes to iSCSI and Fibre Channel is that iSCSI is cheap and easy, and Fibre Channel (FC) is expensive and complicated. This however, isn’t the case.
Disclaimer: This blog simply exists to point out the differences between the two, and the places where each is more likely to encounter pitfalls. Implementations of 1/10 gigabyte iSCSI can work just as well as FC, and FC can be configured badly. We have customers with both implementations who are happy with them.
From a standards standpoint:
Fibre Channel physical topology as well as logical topology and protocols are all defined by the ANSI T11 standard, whereas iSCSI follows 802.3 (Ethernet) for Layer 1 and 2, the IP RFC for layer 3, the TCP RFC for layer 4, and the iSCSI RFC for layer 5.
Since there are so many different independent parts, this can leave a lot of room for interoperability concerns between the different layers as they are implemented by each vendor. Additionally, since these are RFCs, different vendors can implement different versions of the RFC to different results.
From a physical standpoint:
Fibre Channel is run (in modern times) over SFPs with LC/LC cables, or Direct Attached Cables. iSCSI is run on SFPs with LC/LC cables, or Direct Attached Cables (or TwinAX cables), or RJ45 cables with CAT 5, CAT5e, CAT6, CAT6a, or CAT7 cabling.
This opens a lot of room for error when pieces are being purchased since none of these different options will work together directly.
From the vendor standpoint:
Fibre Channel essentially has four players in the market: Qlogic, Brocade, Cisco, and Emulex. This is where iSCSI’s greatest weakness is, its flexibility. Any IP device can technically run iSCSI.
Those ancient 10mbit 3com NICs sitting in a closet? Yep, those will work, as will the old Nortel 10mbit hubs, not to mention that Linksys WRT54g, because believe it or not, iSCSI can go over wireless too. Clearly these not optimal situations, but it highlights one of the biggest issues with iSCSI: because it can be run on anything, people frequently end up — either intentionally or not — with very unfortunate configurations.
From the cost standpoint:
If buying new hardware, 1gb iSCSI will almost always be cheaper than Fibre Channel. If buying new hardware, 8gb FC will almost always be cheaper than 10gb iSCSI. Also, this is not because 10gb iSCSI is faster than 8gb FC. In real world performance, 10gb iSCSI is often slower than 8gb FC due to the excessive protocol overhead mentioned before. An example from a major online retailer, CDW, Cisco will be used for all versions just to come close to apples to apples:
- A 48-port, one-gigabyte Cisco Catalyst 2960s-48: $2,300
- A 48-port, eight-gigabyte Cisco MDS 9148 FC switch 16-48 ports: $5,000+ “+” is used because ports have to be licensed in eight-port increments as needed (after the first 16)
- A 48-port, 10-gigabyte Cisco Nexus 5548u 32-48 ports: $18,000+. “+” is used here because being an Ethernet/IP switch, there are hundreds of licensing options and add on available.
Immediately, anyone who knows anything about these switches would cry foul, “But the Nexus 5548 can do so much more!” which it can; if you pay for licenses. This also leads to the next complication…
From the complexity standpoint:
Fibre Channel HBA/switches are just Fiber Channel. Excluding FCoE (which we are) the Fibre Channel switches are separate and isolated to just storage, all operating the tightly controlled protocol.
Ethernet/IP NIC/switches are handling potentially dozens of tasks. Normal data traffic, VoIP, security cameras, storage and many more tasks. This often means that the Ethernet/IP network is a sprawling environment many times larger than necessary for just storage. It also means that there are a lot of competing items on the network, all needing changes and modifications to the network. All of this leads to many more opportunities for contention, improper configurations, and mistakes.
These are just a few of the many ways in which the two protocols differentiate themselves. There are many more from technical to implementation which we will try to cover in upcoming blogs.