Big Data Enters the Mainstream

In its recent report "Embracing the Big Data Opportunity", the IT market analysis and consulting firm Pierre Audoin Consultants (PAC) states that the hype about 'Big Data' is starting to be matched by reality. Major suppliers turn their attention to Big Data products and services that do not require the specialist knowledge needed today to exploit Big Data sources, while major system integrators (SIs) are building the skills to help their clients embrace the Big Data opportunity.

AC forecasts the global market for Big Data products and services to grow from around 3 billion in 2010 to almost 20 billion by 2016, a compound growth rate of over 38 percent.

Best seen as an extension to the traditional use of BI tools, Big Data analytics enable organizations to deal with large and fast-growing volumes of unstructured and structured data, across multiple sources. Established methods and tools are generally not cost-effective for Big Data information sets.

"We see great interest from many companies, fuelled by a series of reports from consultants and magazines suggesting that major benefits are to be had from Big Data analytics. But several factors have held back the market until today," comments Philip Carnelley, Principal Analyst - Software, at Pierre Audoin Consultants (PAC).

"Supplier push" meets "market pull"

Interest in Big Data has also been driven by vendor marketing, notably EMC, which was early to highlight this market, alongside the rapidly increasing group of Big Data start-ups and specialists, such as DataSift.

But against that, Big Data exploitation has been held back by the need to acquire new skills in arcane technologies - these, while often inexpensive open-source solutions, require rare knowledge sets, which often revolve around the Hadoop data analysis stack, using tools unfamiliar to mainstream developers, like Pig and Hive.

"This situation is changing fast, as the major database companies (analytic and transactional) and BI tool providers enter the Big Data market with tools and technologies linked to their existing offerings. There is something of a trade-off between cheap tools requiring expensive staff effort, and more robust and easy-to-use tools which cost more to acquire - but in general the overall project deployment costs are going down", says Philip Carnelley.

New tools and services will fuel the market

The recent market entry of mainstream providers like Oracle, IBM, Microsoft, SAP and Teradata promise to offer more packaged solutions - "Big Data in a Box". At the moment, they promote an evolutionary approach, building links to their platforms from Hadoop and elsewhere. "Their activity is helping to 'endorse' the market even for more conservative or technically less advanced users, to provide new options", comments Carnelley.

Philip Carnelley therefore advises IT user companies "to look beyond today's difficulties to understand the vendor roadmaps, to see how their developing offerings will intercept with your emerging needs, and plan accordingly. Also, discuss with your services partners how they can help - they surely will have a Big Data practice just waiting for your call."