An Atmos Update

Last week there was some idle banter online that few people knew what was really going on with the Atmos cloud storage platform.  Many people in our industry presume that no news means bad news, and tend to assume the worst.

I knew differently, though.  So I took it upon myself to reach out to some of the Atmos storage team and ask the generic question -- how is it going these days?

And I found the answer far more interesting than I had suspected.

See if you agree ...

A Bit Of Background

The Atmos cloud storage platform was initially announced mid-year in 2008.  Frankly, few people really knew what to make of it at the time.  It didn't neatly fit into established categories, so we found ourselves having to make up a new one: cloud-optimized storage, or COS.

Atmos wasn't just a re-branded NAS filer -- it was a ground up approach to an entirely new use case.  Atmos uses an object-based approach with rich metadata and policy to implement a geographically optimized storage cloud.  EMC offers it in both software-only and integrated software/hardware packages.

If you're interested in my posts on the subject, I'd direct you here.

The good news with Atmos was that it could do things that other approaches just couldn't attempt to do -- it was that radical.

The challenging part is that anytime you offer up something that is truly new, it takes a while for people to figure out what it is and how to best use it.  Sometimes longer than you'd like.  And Atmos is no exception to that rule.

By all accounts, Atmos storage has turned that corner, and is now experiencing dramatic adoption across multple axes.  For example -- I can't share the exact numbers -- but there's a very large multi-petabyte installed base already out there, and it's growing at over 2x year-over-year.  

We also have about a dozen service providers offerings services based on Atmos (more in the pipeline) and a handful of fascinating enterprise-level implementations going into large-scale production.

The Problem Atmos Solves

Let's say you have a *lot* of information sloshing around.  Lots and lots of objects (big or small).  And let's say it tends to be created and used in a variety of locations with somewhat unpredictable usage patterns.

Putting it all in one place creates issues with performance and availability, so you'd like to dynamically spread it around over your network.  You'd also like to implement some policies as well around how the information should move and be protected.

Ordinarily, you might think in terms of the traditional "database plus file system" approach, but that sort of architecture won't easily scale up, out or across.  Or maybe you'd use a bunch of traditional NAS devices and hope the filesystem could figure things out for you -- also not workable.

What you'd really like to be able to offer multiple applications and business users an "information management layer" -- as a service, and do so simply, securely and efficiently.

Whatever you do, you'll need to compete against (and differentiate yourself) against commodity-oriented services from the good folks at Amazon and their brethren.  

That's exactly where Atmos is doing quite well, thank you.

Some Enterprise Examples

There are quite a few to choose from, but I can share a few ones that are already public.

The folks at Ebay, for example, are using Atmos to store and playback vendor-created images that create a richer experience for Ebay customers.  Ebay does business globally, hence the need for a "cloud aware" storage and information management layer that just does this as a service: simply, securely and efficiently.

The folks at CareCore are using Atmos to create a "private cloud" for healthcare documents and images.  Lots of content, sophisticated policies and the need for ironclad GRC.

At the other end of the spectrum, the folks at Vistaprint run a neat business that offers a wide range of "freemium" information-based services to small businesses -- business cards, web sites and the like.  They use Atmos' policy-based mechanisms to deliver differentiated services over and above the free stuff -- and use the same back-end architecture to do both.

They also have a fascinating before-and-after compare of what it cost them to provide this service with ordinary filers, and how Atmos' distributed policy mechanisms not only were far less expensive (storage costs and bandwidth, plus associated opex) but enabled them to deliver differentiated services that were, well, really differentiated.

Starting to get the picture?

Some Service Provider Examples

Last week, we announced at IPEXPO that two of our European partners (MTI and Redstor) had announced new services based on Atmos technology.  These new partners join AT&T, Peer 1 and a passle of other service providers finding business success with Atmos-based services -- they fill a need in the market that can't easily be done any other way.

Competition is being able to do what providers like Amazon can do -- and at roughly the same cost.  Differentiation is being able to do things (through very tailored policies) that these folks just can't do. 

By the way, Atmos environments can easily federate, enabling the potential cooperating service providers to provide global reach as well as local presence as needed.  We’ll probably see more of this going forward -- just like phone networks work today :-)

What About APIs and ISV Adoption?

People familiar with the entire subject of object storage realize that the strength of the model lies in the API abstraction around objects and their metadata, and those APIs in turn aren't of much use if they're hard to use, or -- worse -- nobody uses them.  Indeed, part of the market success of Centera was our ability to get ISVs to use its API to provide differentiated value to our mutual customers.

Generally speaking, the APIs are being nicely adopted on a number of fronts.  First, they're pretty easy to learn and use -- basic REST protocols and web services, nothing too unfamiliar.  Second, we've now got some nice bindings into the Springsource stack, as well as Ruby and a few other popular frameworks.  More info can be found on the EMC Developer Network

We've also been told that the Atmos APIs are meaningfully richer than, say, the generic Amazon APIs.  For example, the Atmos API supports "byte range updates" which are useful when you're only updating a smaller portion of a larger object -- there's no need to read out the entire object, make the modifications, and then read it all back in -- faster as well as cheaper.

You also get a powerful global name space that's sort of built in rather than bolted on.  And more than a few customers have figure out that they can do RESTful calls into and out of this name space with new applications, yet still present selected portions of Atmos as a traditional filesystem mount point as needed to support legacy access models.  

This also lends itself to some interesting workflows where data is touched or processed by multiple applications -- legacy apps can still use a familiar filesystem to access data if needed; newer ones can use the APIs directly.

Atmos adoption was also greatly accelerated by the recent addition of Atmos Virtual Edition -- basically, Atmos software running in a virtual machine.  This allow environments to be developed and thoroughly tested without the need to bring in a bunch of dedicated hardware -- or run production entirely virtual if that meets your requirements.

And, let's not forget, you're welcome to build and run your own Atmos storage cloud, or use any one of our compatible service providers, or any combination of the two that happens to meet your needs.  It's basically the same deal, regardless of how you choose to consume an Atmos-based service.

Are Your Customers (and Partners) Happy?

In a word: yes.

The product has largely worked exactly as expected in terms of performance, security, availability, efficiency and manageability.  The low cost per GB enables Atmos-based solutions to compete and ultimately differentiate against external alternatives as well.  

And we're able to not only roll out new software features on a regular basis (e.g. the recent GeoProtect distributed parity feature) but take advantage of steadily declining storage media costs as well.
Best of all – our Atmos customers and partners are continuing to grow and invest in their Atmos environment as a result – that’s the ultimate measure in many regards.

Is EMC Doing Enough To Get The Word Out?

Probably not.

With Atmos, we've got something pretty cool on our hands: a unique solution to a difficult problem that more and more IT architects are starting to encounter -- the need to create a service-oriented abstraction for large quantities of information objects that's geographically optimized and uses rich metadata to manage policy.  Simple, secure and efficient.

Just about everyone who has tried the traditional "database plus filesystem" approach realizes the inherent limitations, and many have started to look for a better approach.  At EMC, we have the luxury of investing in new groundbreaking paradigms like Atmos, while still continuing to provide more traditional solutions for more traditional problems.

And we probably could do a better job of letting people know that we have something cool that we'd like them to consider :-)

 

Comments

[Pascal]

Hi Chuck,

You should take over Metalogix and push SharePoint+Atmos with StoragePoint. Eyal Sharon alreads wrote about this in his blog.

CX or Symm for the SQL Server, Atmos for the objects. Than IMHO global multi hundreds of TB or multi PB SharePoint installations should become reality i about two years. Who still needs Documentum? :-)

The archiving parts of Metalogix would fit into SourceOne for SharePoint and Exchange.

Best regards, Pascal

[John]

Hi Chuck,

I was trying to look for information about how EMC changed their business model from 2003-2010. I was wondering if you had any knowledge about this subject? Any information about this would be greatly appreciated.

kind regards, John

[Chuck Hollis]

Hi John

We could write several books on that topic alone. Perhaps a better starting point might be for me to ask you -- what specific areas are you interested in?

Please feel free to drop me an email at chuck dot hollis at emc dot com.

Cheers!