Why Big Data & Business Analytics will Take Centerstage in 2014?

With more people spending most of their time online- be it for work, play, learning, or to socialize- the amount of data being generated is truly astounding. Let's dig deep into how Big Data and associated business analytics will gain much traction in 2014.

Fundamental technology disruptions make Big Data attractive for businesses

The notion of dealing with large volumes, variety and velocity of data is not new. In fact, science, defense and intelligence agencies have been capturing and analyzing large volumes and a variety of data for years with proprietary methods and specialized expensive machines. For businesses, there's always a tradeoff to be made between storage of information and more importantly the speed of processing the information collected. The technical advancements in the last few years have broken this impasse. As the chart below shows the cost to store information ($/Mbyte) and cost to process information ($/MFLOP) have been declining steadily. In fact, the cost to sense, collect, generate, and calculate data is declining at a faster rate than cost to store data.



Big Data helps reduce latency "Hot Spots" in  business decisions

Typically, businesses fail to react in real-time to events which impact them. There is latency between events being triggered and actions taken to address those events. There are three dimensions of this latency: First is data latency which is the time taken to collect raw data, prepare and store it and keep it ready for analysis. Second is the analysis latency, the delay in turning data into information and applying business exceptions and rules and generating alerts if appropriate. Third is the source of latency, the decision latency which is the time taken to receive the alert, review the analysis, do a "what-if" scenario analysis and, action.

With Big Data and related analytics solutions, businesses now have an opportunity to address latency hot spots across all layers of their decision making. With in-memory technology, it is possible to shrink the action time dramatically and thus reduce the value loss. All Big Data projects should be viewed from the lens of which of these latency dimensions (one or more) can they reduce.



Big Data makes new signals possible

The traditional measures of organizational performance -balance sheet, income statement, sales pipeline, operations - have been around for decades. In fact, balance sheets and income statements, the two most fundamental measures of business performance, came into existence in the 1930s, before computers were even invented. Today, we live in a completely new, digital world yet the fundamental measures of performance and ‘information tools' we use to improve our organizations have not changed at all. Big Data allows organizations to track new ‘signals' or new measures of performance, new classifications and understanding of their customers, turn new insights into operations, find new measurable ways of doing business that they've never had before. Some of these new signals that are possible are

Brand sentiment - By capturing and analyzing comments on Facebook, LinkedIn, and Twitter, linking them with CRM data to improve customer experience , and optimize campaigns.

Predictive maintenance - By analyzing a continuous stream of machine data diagnostics, predicting the e performance of machinery or likeliness of a break down.

Risk mitigation, real-time - By analyzing financial transactions in real-time, identifying high risk investments, sales, and deals before they actually occur.

360 degree customer view - By storing and analyzing all data about a customer, such as transactions, browsing history, customer profile, social media and a complete view of the customer across all channels can be created.

Asset tracking - Tracking high value assets and identifying abnormal behavior that may put assets at risk of loss, or identifying inefficient usage that is bleeding the business.

Let's now take a look at how to achieve best balance of technologies while managing big data challenges.

Avoiding the free and "me too" trap

When deciding on the best balance of technologies to manage Big Data challenges, some trade-offs cannot be avoided. For example, a lot of Big Data technologies are open-source software, have no license fees, and can run on low-cost commodity servers. However, the total cost of running these clusters can be significant when the numerous servers needed to be managed and the custom development required for the open source solutions are taken into account. In achieving the best balance, it is important to consider the relative performance and costs of different components from open source as well as packaged software and appliances.

Aiming for a ‘me too' Big Data project is not advisable. Although examining what peers in the industry are doing can be a good source of ideas, every business is different. A project that just copies others may not provide the best value. A good way to avoid "me-too" efforts is to assemble a versatile team from parts of the business for a brainstorming session on how Big Data could be used in the organization. This cross-functional team is likely to identify valuable new ways of analyzing combinations of disparate data that Big Data Technologies can handle. For added benefit, including experts from software companies who understand how to leverage Big Data from the perspective of the existing technology landscape is advised

The basic checks

It is critical to identify the data sources that are required for a Big Data projects Estimate data volumes for data sources, including a decision on how long the data will be kept. Understanding to what extent the data is to be processed or analyzed before it can be used is a must. A balance is required between the cost of improving data quality and the benefits it will bring. Identifying the major types of queries, analytics, and processes that will be run on various Big Data technologies and determining if these workloads have peaks and troughs or seasonal variations is important. Finally, checking for any security or data privacy legislations that will influence how Big Data projects can be deployed.

It is about making a difference

The real "so what?" about Big Data is not about data at all, it is about getting a deeper understanding about human interactions, human suffering and making a difference. Whether it is the ability to enhance our understanding of deadly diseases like cancer, improving our global food chain to minimize wastage or preempt terrorist attacks, Big Data is a means to these achieving these ends.

Big Data in 2014

- Cost to store, process information ($/Mbyte) to decline steadily

- Cost to collect,calculate data to decline faster than cost to store data

- Big Data will make new signals possible; Asset tracking, Risk mitigation, etc

- Big Data will help reduce latency "Hot Spots" in business decisions