Yes, We Occasionally Have Some Vigorous Debates

So, sometimes I'm accused of presenting a homogenous view of EMC's perspective.

If I'm doing that, I apologize.  Like any company, we've got our own internal debates on technology, strategy, etc.  Since I write the blog, it's usually my viewpoint that emerges.

So, today, I thought I'd share with you a behind-the-scenes debate sparked by one of my, ahem, more one-sided posts.

The Context

I wrote a bit about the "FCoE debate" recently.  One of my compatriats, Chad Sakac, felt compelled to share his perspective.

Now, Chad is an extremely bright, knowledgeable and passionate guy.  He is certainly a voice to be reckoned with.

But our perspectives are very different.  Chad's primary focus is with what EMC calls "commercial" customers in the US.  I'm exposed to some of that as well, as well as a bunch of other slices of the market.

And the interplay is pretty interesting, if you want to watch:

------- from email -------

Chuck, I love reading your blog always. As often as I disagree with you, I agree - but it never leaves me ambivalent, which always makes it fun.

But this one drove me over the edge and I needed to stop lurking

Chuck says: Nothing wrong with a vigorous debate!

Chad says – you and me both, brother, and I think having this debate publicly is a good thing – it’s highlights that EMC’s views aren’t monolithic, and willingness to show that shows that it’s not just a mouthpiece for marketing.

Chuck says -- can't argue with that

Chad continues:

Representing the mid-size enterprise segment of the market, I can emphasize that iSCSI (along with it's IP storage brother, NAS) is a huge part of our business today, and it's something every customer wants to know more about. I can't think of a customer discussion in the last 6 months where iSCSI wasn't part of the discussion.

Chuck replies:

Absolutely no debate. Couldn't agree more. When you get to the larger enterprises (as I stated) the picture reverses. Bringing up iSCSI in a large enterprise is like farting in church. You just don't do it.

Chad says – agreed.

Chad continues:

For example, we were at Storage Networking World this past week at the IP Storage (iSCSI) Hands-On-Lab, along with Network Appliance and other smaller iSCSI players like EqualLogic, Intransa, Compellent, and HP. At the beginning of each session, people would ask "who wants to do the session for each vendor?" - when it came to EMC, 70% of the audience wanted to try our iSCSI targets in Exchange/SQL Server in VMware-centric use cases. Many of the people who came by where already EMC Symmetrix Enterprise-class customers.

Chuck says: Absolutely no debate in your world. Keep it up, brother!

Chad says: – so far, we’re agreed.

Also, over the last year, I've been involved in many large enterprise customer environments where iSCSI was starting to become a part of their business. The number of enterprise customer discussions around iSCSI is growing exponentially.

Chuck replies:

Hey, all I'm using as evidence is numbers -- our own, and the industry's (IDC). Hard to argue with the numbers.  The picture is consistent: small-to-medium enterprises, iSCSI is extremely popular. Larger shops that started with FC, almost no adoption in large data centers.

Chad says: – more agreement. On the Symm business (a huge part of EMC’s business – the IP ports are included on 2% of deals. That’s a fact. Now, that said, I can tell you right now, the number of cases for me personally where the Enterprise team has done a “reach over” for help is now 2-3 a week for me alone. Anecdotal isn’t the same class of information as statistics, but is still informative. What I see happening isn’t that it’s in the “mission critical” datacenter, but in the periphery, similar with their use of VMware currently (with notable exceptions).

Now - it's indisputable that if you have invested captial investments and human resources in a Fiber Channel network - even if iSCSI or FCoE offered equivalent performance in all aspects - it wouldn't replace the infrastrucutre. Why? For the same reason that Frame-Relay networks still exist. Under some circumstances, the benefit (cost/simplicity/performance) just doesn't outweigh the barriers to change. That's the case on this topic for large customers today.

Chuck says:

Yes, you're right, but there's more. There's politics involved.

Chad says: – no doubt. I always underestimate politics as a technologist.

Chuck says: - even successful technologists should acknowledge politics as a powerful force

Chad continues:

Besides being a classic "innovator's dilema" case, over time that becomes a smaller and smaller Venn diagram over time. I wouldn't want to be in the Frame Relay network business today!

Chuck offers:

Your Frame Relay analogy only goes so far. The people who've deployed FC like FC and see no reason to move -- economic or otherwise. Net new builds separate from the existing network -- sure, they'd consider something different -- but that's not happening a whole lot.

Chad replies: The people who have deployed Frame Relay who still have it like it – but, that’s a shrinking set. Another example – the secret of the RDMBS world – the majority run on DB2 on mainframe. It ain’t sexy, it ain’t growing, it’s not where you place your future bets, but it’s there. FC will be similar , IMHO.

Chad continues from original post:

Likewise, it's indisputable that the opposide is true in the mid-market space. If a customer doesn't have an existing FC infrastructure (captial and skills) iSCSI is a no-brainer. There, iSCSI is growing like mad, and we (EMC) are doing a gang-busters business with excellent solutions and have many many happy customers.

Chuck says: Couldn't agree more. Chad – yup.

Chad continues:

The question is when will the Ethernet-connected block storage value be compelling enough for enterprise customers with huge FC infrastructure investments? For some (not all), it will be with mass adoption of 10GbE (which we support now, but is a rare customer use case). At that point, any enterprise would have too look at the fact that they have two networks with roughly the same functional capabilities and roughtly the same performance parameters (10GbE and 8/16GB FC).

Chuck offers:

In large-scale IT, these are managed (and funded) by different IT groups with different IT responsibilities -- and they don't always get along.

Chad says – agreed 100% -so the question is – who has more “juice”?

Chuck says -- it's not always about "juice" -- it's about what fights are worth fighting, and which ones arent. 

Chad continues:

Likewise, any CIO wouldn't be exercising their responsibilities if they looked at their human resources asking a similar question. Most customers, even the largest enterprises - have IP networking staff that dwarf their FC-SAN networking staff.

Chuck offers:

Yep -- but their skills and outlooks are different. You don't notice it in smaller IT shops. It's very noticable in bigger shops. I could tell you some very funny stories that illustrate the point.

Chad replies – aha – TODAY the skills are different. They aren’t **intrinsically** different.

Chad continues:

Ok - so that's where we agree: "don't bet against Ethernet".

Now - if that train of logic is correct, and we put aside the question of "when with Ethernet-connected storage pass FC-connected storage in revenue" (the only argument is it in 5 years or less?)..... it then comes down to a simple question: will the market sustain two "block storage over Ethernet" standards?

I think the reasonable conclusion is... NO. There are almost no mass-market standards that serve the same function (let's see the HD-DVD vs. Blu-ray state in a few years)

Chuck says:

Yes, they will, because many of us believe it will be a single standard. The proposed ethernet chip for FCoE also supports IP stacks, etc. FCoE (and DCE) becomes a superset of the 1Gb/10Gb world we see today.

Actually, it's pretty interesting when you look at it. The idea is to create "virtual channels" using 10Gb enet. One channel could be used for IP-oriented connections. Another channel could be used for RDMA clustering where latency matters. And another could be used for FCoE. All at the same time.

Back to DVDs, players that support both at no incremental cost makes it a different discussion.

Chad says – no doubt, if the cost is free.

Chuck continues:

That's what the chip vendors are telling us their building. So, it's more of a question of "what combination of network paradigms do you need?"

Chad says – if it’s free, then the ASIC vendors will include it (and of course, the incremental cost is truly free – the manufacturing cost to an ASIC vendor is near zero when you have the Intellectual property blocks laid out).

Then the next logical question becomes: "which will it be then - iSCSI or FCoE"?

Chuck continues:

In this world, it's not a mutually exclusive choice. If the hardware supports both concurrently, which would you choose?

Chad interjects – right, so let’s even concede that it’s FREE from a hardware standpoint, and on **every** interface card (so it looks like a NIC/HBA/RDMA all at once – there are others doing this with infiniband – check out xsigo).

The problem is not the ASIC vendors. There is a dramatic fan out of what the “ecosystem” (OS, drivers, software, training) needs to do with any technology –so it’s not free. Then the question becomes one of “is the benefit enough to outweigh those factors”. Let’s do an 10GbE iSCSI test and an FCoE test. If the difference is 2x or more, I’m open minded. If it’s not – that’s not enough to have the market want it. One of my favorite quotes: “Technological change is inevitable – if it isn’t done by incumbents, new emerging players will do it. If it isn’t done in a regulated industry, new unregulated industries will emerge.” – Andy Grove. What happens if, for example, MS says: “yeah – not interested” – not saying it’s going to happen, but just think about it. It shows that it’s more than the ASIC folks question.

Chuck says: ecosystem adoption is a factor of any new technology.  The question is less about the vendors, it's more about customer demand.  And, in certain parts of the market, customer demand for FCoE seems pretty high.  We'll see.

Chad continues:

The answer is simple. iSCSI. iSCSI will has been building an ecosystem of support for the last 6 years. Microsoft has an incredibly robust iSCSI initiator. Linux isn't quite as solid (particularly on the native multipathing side), but it's sure getting there fast. VMware's iSCSI initiator likewise is excellent - and frankly has more robust multipathing options than FC. Every major vendor will have had iSCSI targets (of varying capability) on the market for years.

Critically - iSCSI is **good enough**. The performance of iSCSI on GigE networks for our mid-range customers is already often more than enough. 10GbE will make this better - literally an order of magnitude better.

Software initiators are perfect for todays quad-core world, and only get better in tomorrow's octal-core world.

Chuck continues:

You want a storage network that looks like IP? You want to route things? You'll be an iSCSI fan.

You want a storage network that looks like FC? You want latency and deterministic behavior? You'll be an FCoE fan.

But you'll end up with one set of HBAs, one set of switches, and one set of target connections to do both or either.

Chad disagrees:

Will FCoE be marginally more efficient protocol for block over Ethernet than iSCSI? Sure. But those IP networking folks - won't they prefer something they know and have been working with for years? Won't they prefer something with broad OS support? How much better would FCoE need to be to overcome that? 4 times better, or the usual "order of magnitude" better? What about cost?

Chuck says:

(Getting tired) By now, I think I've made my point.

Chad says: I disagree, I think I’ve made my point. My point is that even if it is free from a hardware standpoint, and you can do both – the market will CHOOSE one, unless the difference between both is significant. This has happened time and time again in the technology business. This is why standard wars happen – win the standard, you win the business. The war of “block over Ethernet” has been happening for years, with only one party showing up.

iSCSI - good enough today, good enough tomorrow. FCoE just won't survive. Let's not be Sony and ride BetaMax until we auger in just because we like it intellectually.

Chuck says:

I don't see any reason why the market would not need both.  The concept of "winner" here is elusive.  I know large enterprises who will never go iSCSI in any meaningful way, but are interested in FCoE.  And I know not-as-big enterprises who are more than happy with iSCSI, and see no need to do anything different.

Chad says:

I need to get a soapbox like yours

Chuck says:  why not?

Your doppelganger - Chad

[note: light editing was done throughout to improve readability.  I don't think I misrepresented Chad's position in the process of doing so -- Chad?]

----------- end vigorous email debate -----------

Now, if you've made it this far, this is a pretty interesting debate.

And it's not unusual. 

The nice thing about EMC is that we can have these discussions -- no harm, no foul -- and come away with a different set of perspectives.

And that's just plain fun ...

Comments

[Chad]

No edits changing meaning - and in the end this debate will be won one way or another.

What do you say: market share split of iSCSI vs. FCoE by 2012. I bet you a bottle of fantastic wine (recently had and enjoyed Dusky Goose, Pinot Noir, Oregon - 1999) that iSCSI takes it by a factor of 10.

[Chuck Hollis]

I'd take that bet.

But if you want to up the stakes, you should know that my wife collects vintage Bordeaux.

I, for one, can't really get my head wrapped around a four-digit bottle of wine.

[Chad]

I guess that's why I'm all about iSCSI and you like FCoE....

Just like when it comes to wine, I like a good value - great wine/great price/simple and easy to use (buy it at Trader Joe's).

You live in the stratosphere :-)

I'm so confident - I'll take your bet on your terms!

Chad

[the storage anarchist]

At least one of you seems to have overlooked that FCoE is not FC over IP – it’s simply FC protocol over the CDMA signaling of Ethernet on standard CAT6 (or CAT8) cables - with only the slightest of additional overhead added. iSCSI, on the other hand, is in fact SCSI over (the notoriously heavy) IP protocol.

And that subtle little difference drives a BOATLOAD of performance, throughput, latency and overhead difference. No matter the adoption curve of iSCSI, a significant percentage the FC world will want FCoE because it’s as fast and low-latency as InfiniBand, but uses a proven protocol over an inexpensive and well understood cabling infrastructure…and iSCSI will undoubtedly be used to serve the rest of the data center that doesn’t have to worry about latency.

And yes – the same hardware will support both, and for a single price. The iSCSI stuff will be easier to deploy because it’s readily routed and configured, and while customers will accept the higher complexity of managing FCoE over the glass-house DCE cable plant JUST TO ACHEIVE THEIR LATENCY REQUIREMENTS.

I say this somewhat authoritatively, because this demand for low-latency/high throughput is an ever-increasingly significant portion of my customer discussions for Tier 1 (or more appropriately Tier 0) storage. In this space, there is an emerging URGENT unmet need for I/O response times previously unimagined, and well outside the practical realities of IP-based connectivity.

And I don't think you should be measuring the "winner" by market share...there's room (and need) for lots of different connectivity options out there - one size need not fit all.

Now, if there's a way I can get into this Bottle of Bordeaux Bet without sacrificing my position (or having to accept a volume-based definition of "winner"), I'm terested...

[Chad]

anarchist, I know exactly the difference - and understand the downsides (and upsides) of being up one level in the stack.

Performance is one of many things that drives what customers do.

The market is big enough, and customers in different segments are different, and EMC is broad enough that we can serve them all.

But - that said, I'll **STILL** double down - take you in on the original terms.

Heck, I'll even make side bet with you (Chuck and I have spit and shaken on our terms) - define what you think would be FCoE market success, I'll define what I think would be iSCSI market success - and we'll measure against the same yard post - 2012?

On the performance side - iSCSI's "notoriously" bad performance? Good enough for many. I'm ready for a 10GbE bakeoff today - iSCSI vs. FCoE - you fellas game?

[Storagezilla]

As someone who spends nearly all of their time in pre-sales I can say with certainty that EMC will sell whatever there is demand for.

You guys can fight it out over the tech but if our customers require we use silly string as the transport you're all going to help me sell it to them. :-D

[the storage anarchist]

the anarchist's 2012 market share blognostication (by port count):

NFS/CIFS over Ethernet: 40%
iSCSI over Ethernet: 25%
iSER/SRP over IB: 2%
Native Fibre Channel/FICON: 10%
Fivre Channel over Ethernet: 20%
Other (CAS, etc.): 3%

Translated into raw tranport:

Ethernet: 88%
FC: 10%
IB: 2%

Now, by REVENUE, things will likely look a little different...the low-latency market is clearly willing to pay more to get what their applications require, while the "IP is good enough" market will be commoditized to the point that the ports are essentially free (and possibly uncountable).

[TechEnki]

A couple newbee questions:

- Does FCoE assume using a single switch for standard Ethernet and FCoE (it sounds like it does)? If one barrier to iSCSI adoption in the enterprise is the dislike in merging network management between the network and storage groups, FCoE will have the same barrier. Who would own/manage the FCoE SAN?

- FCoE relies on an enhancement to Ethernet that gives a similar loss-less capability to Ethernet as exists in FC. Since it is at OSI layer 2, it could be applied to IP connections to make them more deterministic in local (non-routed) environments as well. If I assume full hardware optimization for both iSCSI and FCoE on the same enhanced Ethernet layer, do we know what the latency and performance differences will be? Will iSCSI catch up or does the FC protocol retain the advantage?

- Does FCoE require hardware (ASIC) optimization or is that optional?

thanks for any insight,

TechEnki

[NoHype]

If the bet can be FC+FCoE vs. iSCSI, I think I'd jump on that. If you take the Anarchist's revenue, in 2012 it still comes out on top. IDC and others get paid forecast these markets and their hit rate on anything further than the next 1-2 years is not as good as the farmer's almanac is at the weather. It's EMC's blessing and curse that we get to play in all of the markets due to the product portfolio and breadth of customer needs.

[Fred San]

The advantage of FCoE IS that it's not a routable protocol. Just stop and think, do want the FC packets going out over LAN? even a gateway or router? The packets need to stay within the data centre. Like AoE (ATA over Ethernet) it is a poor man's FC and I'm poor so I like it a lot. BTW, Linux has had support for AoE for some time and there are some excellent drivers out there for Windows and Mac OS/X. See www.coraid.com.

Interestingly nobody has commented upon the difference between the costs of Brocade switches v. 10G ethernet switches. To me that together with the NIC v. HBA cost advantage will be a driver.

[Amnon Izhar]

Seems this bet is about to be determined. I think we all know where FCoE ended up compared to where we expected it to be circa 2007