The Rising Affection Towards Big Data Mining

– Madhusudhan KM, Chief Technology Officer, Mindtree

BigData adoption is on the rise. Most enterprises either have adopted BigData solutions or are on the verge of implementing it. BigData solutions kicked off as a phenomenon for low-cost storage and massively parallel computing in batch mode. Today, these solutions are capable of performing real-time analytics using techniques such as streaming analytics coupled with deeper data mining techniques. Following are some of the key trends we foresee in 2016.

Enterprise Data Lake – Organizations have spent decades integrating silo data sources and continue to face challenges in terms of deriving value by combining heterogeneous data sources (both internal and external). Inability to consistent access of transactional and large historical data poses a challenge for data analysis techniques. Data lake addresses this challenge by bringing silo data sources (structured, unstructured and semi-structured) under one umbrella (Hadoop like ecosystem). Data lakes offer unified data management capabilities in terms of metadata management and auditing. Due to the gaining popularity of data lakes, many public cloud providers are now providing Data Lake as a PaaS offering. Data security is another area where we see a rise in new sophisticated tools enabling advanced encryption and data governance mechanisms.

Madhusudhan KM Chief Technology Officer, Mindtree

“BigData platforms leverage multiple tools/technologies for batch/micro batch processing, real time analytics, machine learning, graph processing etc. This complicates the IT landscape, Support and operations of BigData platforms have increasingly become challenging.”

IoT – Big Data and analytics are an integral part of Internet of things. With the rising number of devices and smart sensors (in cars, building, cities, manufacturing plants, wearables, etc.) exabytes of data get generated every day. IoT solutions will leverage Big Data capabilities like streaming analytics, complex event processing in real-time and NoSQLs to store time series data. Adoption of analytics at the edge (fog computing) is picking up to enable proximity computing. Fog computing enables local analytics to perform quick real-time decision-making before sending data onto the cloud.

Deep Learning – With a rise in the amount of data and data diversity, it is becoming increasingly difficult to apply prebuilt models for machine learning. Hypothesis validation is becoming a cumbersome task. Deep learning based on artificial neural network techniques is being used to identify patterns, predictions without applying pre-built models. Large corpus of data is essential for these self-learning techniques to accurately predict the outcome. Deep learning techniques are being used for image processing, scene detection, predictive modeling, etc. Deep learning is expected to make a significant contribution to text analytics and image to text semantic generation.

New age BigData platforms - Big Data platforms leverage multiple tools/technologies for batch/micro batch processing, real time analytics, machine learning, graph processing etc. This complicates the IT landscape and support and operations of Big Data platforms becomes  increasingly challenging. New age Big Data platform like Apache Spark are gaining popularity as it brings different type of workloads under a common platform. In memory analytics, capabilities make Spark perform better in comparison to traditional MapReduce programs. We expect enterprises to adopt new age Big Data platforms to simplify the technology landscape.

Data Visualization – Quest for becoming a data-driven organization is driving enterprises to adopt data discovery and visualization tools. Apart from traditional enterprise reporting tools, organizations are expected to invest heavily on data discovery tools that enable business users to freely explore data. These tools also aid decision science by helping data scientists to easily identify feature metrics essential for machine learning techniques.

In summary, BigData solutions open up enormous opportunities to build solutions that were otherwise never possible to build.