Chuck's Blog

From Big Data To Modest Data

Mea culpa. 

Several years ago, I became thoroughly enamored with the whole notion of Big Data. 

I, like many, saw a brave new world of petabyte-class data sets, gleaned through by trained data science professionals using advanced algorithms — all in the hopes of bringing amazing new insights to virtually every human endeavor. 

It was pretty heady stuff -- and still is.

While that vision has certainly is coming to pass in many ways, there's an interesting distinct and separate offshoot: use of big data philosophies and toolsets — but being applied to much smaller use cases with far less ambitious goals.  

Call it Modest Data for lack of a better term. 

No rockstars, no glitz, no glam, no amazing keynote speeches — just ordinary people getting closer to their data more efficiently and effectively than before.

That’s the fun part about technology: you put the tools in people’s hands, and they come up with all sorts of interesting ways to use it — maybe quite differently than originally intended.

Paul Maritz Explains Pivotal

One of the more memorable sessions at this year's EMC World was when Paul Maritz took the stage right after Joe Tucci to walk through the rationale and motivations behind Pivotal.

I'm sure more than a few people walked into the keynote session wondering what this Pivotal thing was all about.  Not only did Paul do a masterful job explaining the concepts, he made the compelling case that a platform like Pivotal must exist.

The entire video appears here; Paul starts at 32:00 or so. Very much worth watching in its entirety if you have the inclination.

For everyone else, I thought I'd attempt to summarize some of the core ideas in an easy-to-digest form -- although Paul is a hard act to follow.

GE Invests $105 Million In Pivotal

Occasionally, we get very clear signs that significant change is in the air.  When large sums of money unexpectedly change hands, people take notice that something interesting and perhaps unexpected is happening.

This morning, GE announced that they were investing $105m to take a significant stake in Pivotal, the new initiative jointly owned by EMC and VMware.

On one hand, we have GE: perhaps one of the best examples of an exceedingly well-run global corporation with a market cap of over $220B.  On the other hand, we have Pivotal: a nascent analytics platform company formed from EMC and VMware assets, with an exceptional leader at the helm: Paul Maritz.

Why would a well-resourced and exceptional global corporation take a significant stake in what might appear to be a technology startup venture? 

And what might this signal going forward?

Hadoop Is Growing Up

For many, Hadoop has become much more than just an interesting data management technology -- it's starting to be seen as the potential strategic successor to the familiar relational database world. 

It's the platform where the next generation of big data applications will be built.

Along those lines, EMC's Greenplum division -- now part of the nascent Pivotal Initiative -- has announced a considerable suite of extensions and enhancements to the increasingly popular Hadoop environment. 

But please don't get lost in the product detail -- there are some very big and potentially controversial ideas in play here. 

The role that Hadoop is now playing in big data analytics is now unquestionable; the race is now on for who can provide the best enterprise platform and ecosystem.  And, from where I sit, the new Pivotal group is doing a stellar job of earning that title.

With this announcement, there's a new high-performance SQL layer to make more data available to more data workers. 

There are new capabilities to parallelize analytics processing.  There's now a meaningful set of integrations with VMware and virtualized infrastructure.  There are new consumption options for infrastructure and especially storage.

And there's a new set of management tools aimed squarely at the administrators of these environments. 

All of this wrapped up in a new name (Pivotal HD) that reflects the new direction of the team.

Hadoop is growing up -- fast.