I Want My pNFS!

From time to time, various vendors attempt to combine the best of the two most popular storage access models in our industry: block (presumably on FC, iSCSI or FCoE) and file (most always on ethernet/IP).

Historians of our industry can point to many brave but ultimately unsuccessful attempts to combine the inherent simplicity and ease-of-management of NFS with the performance and availability attributes of SAN.

Many of us have been tracking the progress of pNFS in this regard, and it looks like the time is nearing when customers can seriously consider enterprise-class implementations of this converged access model.

The big question in everyone's mind: will it permanently blur the lines between file and block access?

To Begin With

It's hard to argue with the utter simplicity of file systems as a way to organize and present storage.  Everything has a name, the names are usually logical, they're hierarchical, they can be easily redirected and abstracted, and so on.

All of us computer types cut our teeth on one type of file system or another, so we intuitively seem to know our way around.

Block and LUN SANs, however, take a bit more getting used to.  The names are less logical, there's little or no hierarchy, familiar filesystem concepts and operations can't be used, and so on.  But -- historically speaking -- they've tended to deliver superior response times, bandwidth, predictability and availability.

The Best of Both Worlds?

Any storage type familiar with both has always wondered -- would it be possible to combine the presentation and access model of NFS, and deliver the performance, predictability and availability characteristics of SAN?

Many companies have taken a stab at this at one time or another, including EMC.  If you're familiar with MPFS, it was an extension to the NFS environment that used the NAS servers mostly as a metadata server -- and then used SAN protocols to directly access the data itself.

The good news?  It delivered the intended benefits -- we have great stories of customers building large enterprise "SANs" (using FC at the time) that were accessed and managed as NFS (using ethernet).

Although the setup could produce eye-popping results, there were two inherent challenges.  

First, at the time, you were looking at two separate I/O subsystems --- one for file protocols (ethernet/IP) and a separate one for block protocols (FC SAN).  That could get both expensive *and* complex.

The second challenge was that we were the only ones really doing it at the time.  You had to use our server-based MPFS client, only use EMC NAS, etc.  No real ability to freely mix and match.  That's an obvious inhibitor.

And with the pending availability of pNFS (aka NFSv4.1), it looks like both challenges are being addressed.

What's Different Now?

First, it's no surprise that 10Gb ethernet is both a technical and economic reality for many.  Whether you choose to run IP protocols (e.g. NFS and/or iSCSI), FC protocols (e.g. FCoE), -- or some combination -- we're at a point where we're not looking at multiple flavors of physical storage connectivity to make the scheme work.  It's ethernet -- period.

Second, we've now got a workable industry standard that most every vendor seems interested in implementing.  It's evolutionary, not revolutionary.

The core concept in pNFS 4.1 is the notion of a "layout" which clients request from storage devices.  At an extremely high level, the layout is nothing more than a map of logical objects to more physical locations.  Layouts can be granted, updated, revoked, etc.

A client -- using the layout -- can now directly access storage objects without the metadata servers (e.g. NAS head in this example) getting involved.  That's a big win, when you think about it.

If you'd like a more detailed overview on all of this, please take a look at this summary deck from Sorin Faibish of EMC who is our lead on this initiative.

And On To The Bake-A-Thon!

At some point, every vendor working on this stuff needs to get together in a single location, and try out their interoperability with every other vendor.  Call it a plugfest, call it a bake-a-thon, call it whatever -- it's a key step in making any interop standard a reality.

Coordinating and staging such an event is no easy trick, though.  It takes a considerable amount of time and effort, and is clearly an area where the larger vendors have a role to play in moving things along.  As an example, EMC is proud to be sponsoring just such an event next week.  If you're a vendor -- and you have some interest -- the flyer is here.

Note to all: us vendors do occasionally cooperate behind the scenes when it's important to our customers.  Widespread vendor participation in this particular pNFS 4.1 event is a strong indication of just how broad industry acceptance might be going forward.

The VMware Angle?

I don't want to speak for VMware regarding their intentions for integrating pNFS 4.1 support as an integral part of the ESX I/O stack, but -- you have to admit -- being able to use a single, standardized NFS-type file system that had the I/O characteristics of a block-oriented world -- well, that's a juicy scenario to consider.

And, since the metadata server isn't necessarily in the data path, one can imagine smaller, virtual machines being the "NAS head" with all the heavy I/O being done direct from server to storage with very little in the way.

Convergence Is King

In our industry, convergence is a hot theme: convergence of function, convergence of infrastructure, tighter integration of technologies with others in the stack, and so on.

The idea of a converged storage access model, combined with a converged storage network, accessing a device with converged functionality and architecture -- well, that's just too cool to pass up.

 

Comments

[Chuck Hollis]

Chuck here ...

Sorry, screwed up the link to the PPT deck. Should be fixed now. Apologies to all ..

-- Chuck

[Chuck Hollis]

Chuck here again, sorry ...

I neglected to point out -- but a few of you picked up -- that the "layout" concept is not restricted to just file and block -- there is a clear architectural pathway for object-oriented models as well.

Having said that, maybe we're looking at an entirely new definition of "unified storage"?

-- Chuck

[Alan Newman]

Chuck,

I'm with SwiftTest. We attended the CIFS/SMB Plugfest last week coincident with SDC, and we would like to participate in future "Bakeathons". Many of the participants listed for the Bakeathon are our customers- including EMC in multiple locations.

We're focused on building test tools for unified storage systems. We currently support CIFS/SMBv1 & v2; NFSv2, v3 & v4; iSCSI & HTTP/Object Storage. We'll have NFSv4.1 and FCoE in the first half of 2011.

The industry is moving from product silos to unified storage, and our products replace testing silos with consistent testing practices across all protocols. I know unified storage is important to EMC, and we want to help. :-)

I'd appreciate the opportunity to share our product strategy with you.

Please let me know your availability over the next couple of weeks.

Thanks.

Alan Newman
VP Marketing
SwiftTest
anewman@swifttest.com
408-218-4134

[Sorin Faibish]

I want to provide some additional technical background to the Bakeathon and pNFS maturity.
pNFS is a new NFS protocol that allow high parallelism in access multiple NFS servers in parallel with very high in and out scalability. The pNFS protocol is the first that introduces the notion of layout which is a map of the file's blocks location on the network file
system. There are 3 flavors of the pNFS protocol differentiated by the type of the layout and backend storage: file layout (RFC 5661), block layout (RFC 5663) and object layout (RFC 5664) corresponding accordingly to 3 types of backend storage; NFS file servers, LUNs
in storage arrays for block and object based storage (similar to Centera) for object layout.

The intention of the pNFS protocol is to unify different types of storage and cluster them in a single networked FS. This is the standard answer to a multitude of proprietary clustered FS that made users dependent on the
vendor/technology of choice: Lustre, GFS, PVFS, OFS, IBRIX, Polyserve and many others. pNFS will allow users to combine different storage technologies in a single distributed storage system and allow wider choices and lower prices for the IT departments.

This is an opportunity for EMC to showcase leadership and prove one more time that standards are crucial to EMC success. The Bakeathon is an interoperability engineering plugfest event that allows different vendors and users to validate the implementation of their NFSv4.1 and pNFS servers and clients. EMC has funded
CITI (Center for Information Technology Integration at University of Michigan - Ann Arbor) to develop pNFS block client for Linux for the last 5 years resulting in the inclusion of the pNFS block layout in Fedora 13. Peter Honeyman the manager of CITI will be attending
the event as well.

There are 3 such NFSv4 plugfests events each year. The event has increased importance for EMC this time around, as EMC was the first vendor to announce the official GA of the first commercial NFSv4.1/pNFS server (in Unified
Barossa.) EMC has the intention to become the permanent home of the Fall Bakeathon replacing SUN's previous sponsorship.

EMC has played a central role in pNFS arena as founding members and co-chairs of the SNIA ESF as well as members in the NFSv4 working group in IETF. EMC donated IP of MPFS to the IETF community and owns the block layout protocol RFC 5663.

[Sorin Faibish]

Forgot few points:
Although at this time there are only 3 types of layouts the pNFS protocol leaves door open to any additional new layout types and storage technologies to be added in the future. For example there are already patches that will allow pNFS support for Lustre.

[Mike Riley]

For a little more background on this topic, I'd suggest visiting Mike Eisler's blog. Mike co-chaired the working group for NFS4.1 and pNFS and has been editor of the spec along with Dave Noveck of NetApp. Here's an early article on what pNFS will buy you and, fortunately for me, included illustrations!

http://www.netapp.com/us/communities/tech-ontap/pnfs.html

Also, Mike has some history of the spec as it weaved its way through the standards process:

http://blogs.netapp.com/eislers_nfs_blog/

Lot's of good info there in addition to what you have here.