Chuck's Blog

Hadoop Is Growing Up

For many, Hadoop has become much more than just an interesting data management technology -- it's starting to be seen as the potential strategic successor to the familiar relational database world. 

It's the platform where the next generation of big data applications will be built.

Along those lines, EMC's Greenplum division -- now part of the nascent Pivotal Initiative -- has announced a considerable suite of extensions and enhancements to the increasingly popular Hadoop environment. 

But please don't get lost in the product detail -- there are some very big and potentially controversial ideas in play here. 

The role that Hadoop is now playing in big data analytics is now unquestionable; the race is now on for who can provide the best enterprise platform and ecosystem.  And, from where I sit, the new Pivotal group is doing a stellar job of earning that title.

With this announcement, there's a new high-performance SQL layer to make more data available to more data workers. 

There are new capabilities to parallelize analytics processing.  There's now a meaningful set of integrations with VMware and virtualized infrastructure.  There are new consumption options for infrastructure and especially storage.

And there's a new set of management tools aimed squarely at the administrators of these environments. 

All of this wrapped up in a new name (Pivotal HD) that reflects the new direction of the team.

Hadoop is growing up -- fast.

Greenplum: Pushing The Boundaries Of The New Data Science

By now, the storyline ought to be familiar.

Around the globe, leaders in both the private and public sector are clearly besmitten with the amazing power of big data and predictive analytics.

Much has been written so far, with more coming -- two of my favorite pieces are here and here.

In this new hegemony, the ascendant rock star is the data science professional, or -- if you prefer -- the data scientist. Much has also been written about this realtively new and essential caste of knowledge worker -- for example, this piece I did a while ago.

EMC, through our Greenplum division, is "all in" on this fundamental shift on how data is being used to create new insights and new value.  That's a rather bold position, as I still routinely encounter people who are still firmly entrenched in the comfortable and familiar world of data warehousing and business reporting, and can't understand what all the fuss might be about.

But I'm not despairing.  I meet more than enough people around the globe who are extremly  motivated by the unfolding new world of big data, predictive analytics and data science.  And these people are likely to be paying close attention to what  Greenplum is doing.

From enabling technology to investing in the data science communit Greenplum is among a small handful of vendors actively bringing the future into the here-and-now for so many of our customers.

And with today's announcements, you'll see clear evidence of that strategy in action: more investments towards building the industry's most powerful ecosystem for exploiting big data analytics.

VMware's Project Serengeti And What It Means For Enterprise Hadoop

Virtualize something -- anything -- and you make it easier for everyone to consume: IT vendors, enterprise IT organizations -- and, most importantly, business users.  The vending machine analogy is a powerful and useful one.

At a macro level, cloud is transforming IT, and virtualization is playing a starring role.

Enterprise-enhanced flavors of Hadoop are starting to earn prized roles in an ever-growing variety of enterprise applications.  At a macro level, big data is transforming business, and Hadoop is playing an important role.

The two megatrends intersect nicely in VMware's recently announced Project Serengeti: an encapsulation of popular Hadoop distros that make big data analytics tools far easier to deploy and consume in enterprise -- or service provider -- settings.

And if you're interested in big data, virtualization, cloud et. al. -- you'll want to take a moment to get more familiar with what's going on here.

VMware To Acquire Cetas: Making Analytics Easier To Consume

Today VMware made an acquisition that's extremely interesting from several perspectives: cloud, virtualization stacks, big data analytics and more.  

While certainly no Instagram, I think we'll all look back in a few years and agree -- yes, this was an important one.

Cetas describes themselves as ".. an early stage startup that delivers Analytics-as-a-Service to customers and is driving the democratization of analytics and business intelligence by dramatically lowering the barriers to entry for all companies, from SMB to large enterprises."

Yes, quite true when they were a standalone entity.  

But as part of the VMware (and, by extension, the EMC) family, it gets much more intriguing indeed ...

HDFS: Coming Soon To An Array Near You

As enterprises around the globe are beginning to exploit the value in their unstructured data, Hadoop is starting to find its way into more enterprise settings.

Enterprise IT buyers are justifiably a demanding lot, which creates plenty of opportunity for IT vendors such as EMC to create products that meet their newer needs.

Such is the case with the new native HDFS (Hadoop Distributed File System) capabilities from EMC's Isilon group.  

As you'll see, it makes exploiting the power of Hadoop and large unstructured data sets incredibly more efficient and powerful.

And before too long, I suspect other storage vendors will scramble to offer something very similar :)

Analytically Enabling The Enterprise via BI As A Service

Many amazing things being done in our EMC IT group these days.

The original premise was that -- once we had made enough progress on transitioning to an ITaaS model -- the value-added strategic IT initiatives would start to come hard and fast.  Well, they are.

Why?  

Almost everything IT does now is based on variable, re-usable (and competitive!) services: IaaS, PaaS, SaaS etc.   IT friction is dramatically reduced as a result.

Business users are now marketed to, and not simply told what they can and can't have.  

A financial governance model based on price signals and an internal marketplace is now starting to optimize IT decisions.  

It's a very different IT game here at EMC than just a few short quarters ago.  Perceptions have changed; enthusiasm is everywhere.  And I get the privilege to share it with all of you ...

In a previous post (The Journey To Big Data Analytics), I outlined the process by which most companies we thought would likely acquire and leverage new analytical capabilities.  At a high level, we now feel prepared to start our own internal journey in earnest, much like we're simultaneously tackling the challenge of mobilizing our extended enterprise.

In this post, I'd like to begin to share with you how we're organizing for success around this particular topic.  Yes, we're using plenty of EMC technology (naturally!), but the real key to anything transformative is the organizational approach.

The Journey To Big Data Analytics

Our story so far ...

Many now realize that big data analytics is the next big thing in creating value from information. 

As such, more than a few people are wondering how they might think about their journey: how they get from where they are today to where they want to be in a sequence of logical, digestible and justifiable steps.

We spent a great deal of time in 2011 learning from organizations who had visibly achieved this proficiency.  What kinds of skills were required.  How they were organized.  And, more importantly, how they got to where they are today.

We've boiled down our learning into a relatively simple three-phase model that describes that pathway that many organizations seem to follow.

And, if you're wondering how to get started on your own journey, perhaps you'll find this useful.

Greenplum UAP And Enabling Your Data Science Team

It must be Data Science Week here at EMC.  

I've already written about our latest findings from the EMC Data Science Survey, and shared how EMC Education is offering their first Data Science Associate coursework to support the increased demand for knowledgeable individuals.

It's almost inevitable that all of this would be leading up to an EMC product announcement, and -- yes -- that's the case.  

The new Greenplum UAP (unified analytics platform) is especially exciting to me because it appears to be the first integrated software suite that is precisely targeted at the newer style of data science teams.

Here's the pitch: in the new world of big data science analytics, all the value comes from the data science team.  Since it's pretty obvious that (a) knowledgeable data science professionals will be very scarce resources for the forseeable future, and (b) their time is extremely valuable, then ostensibly makes sense to build tailored software environments that maximize their productivity.

That, in a nutshell, is what makes Greenplum UAP so unique and compelling.

More Big News From Greenplum

I love these guys. 

It wasn't long after their acquisition by EMC, I realized that -- not only had EMC acquired state-of-the art database technology -- but we'd acquired a world-class team as well.

Shortly after the acquisition, EMC announced the formation of a new division -- the Data Computing Division with Greenplum at its core -- and its first hardware platform, the Greenplum DCA -- data computing appliance.

This was shortly followed by a popular "community edition" of their powerful platform and toolset  Their full toolset, basically free for non-production environments.  More coolness.

And today, at the SAS Global Forum, they did it again: announcing not only a major integration with SAS, but entirely new versions of the DCA as well. 

Impressive.

Community Tools For The New Data Scientist

A seriously cool announcement from the EMC Greenplum folks -- a new "Community Edition" of their powerful database, as well as related open-source analytical power tools to put all that big data to work. 

All free.

I had a serious fling with analytics a long time ago.  Maybe it's time to rekindle that old flame :-)