As the Internet of Things (IoT) starts invading the world around us, the stage is set for an ‘Analytics of Things’--a utopian world where big data analytics is not confined merely to solving business problems, but also upending the way business gets done.

Even before data started getting recognised as the ‘new oil’, enterprises outlined the need for a system that could handle the onset of ‘information explosion’. In 2006, Hadoop was born as an open source distributed storage and processing system spread across commodity hardware. The Hadoop framework led to the incubation of a large number of open source projects targeted at various aspects of data and analytics.

The initial focus of Hadoop as a big data platform was towards storing large volumes of data in a cost-effective way. The adoption approach of big data was majorly ‘experimental’. Enterprises were not sure how to go about the task. The foremost requirement was to ‘dump’ all the data in a big data lake and figure out use cases later. The big data architectures had very little diversity with a standard set of tools (largely open source) responsible for data management.

Most enterprises also grappled with the coexistence of data warehouses and big data lakes. Telecom firms and banks were among the first to move towards big data. Prior to stepping into big data, they were primarily driving their reporting systems through data warehouses with a variety of data management tools. Some of them moved their entire reporting systems to Hadoop, while others chose to treat Hadoop as a ‘secondary’ storage and split logic across the two systems.

Over time, the universe of big data tools witnessed sharp growth. This led to enterprises implementing a new breed of use cases like recommender systems, fraud detection, digital marketing automation and supply chain optimisation on big data utilising the power of real-time processing hubs. At the same time, companies also had their concerns on the complexities inherent in big data implementations.

As a result of these demands, big data architecture entered its first transformation. Hadoop evolved from phase 1.0 of its life to phase 2.0, with improved speed and real-time analytics. During this time, architectures became more diverse. In addition to enterprises moving their analytics stack to big data, the other major driver for this diversity was the emergence of Platform as a Service (PaaS) components from different cloud vendors. This diversity was a much-needed answer to enterprises moving towards a ‘fit for purpose’ architecture over the generic nature of Hadoop components.

As companies further explored extracting more value from their big data investments, it led to the rise of two major trends in big data: data monetisation and Analytics as a Service (AaaS). In both these aspects, the big data industry experienced yet another revolutionary transformation. The AaaS platforms were powered by domain experience and provided the much needed context to generic PaaS platforms. The big data architecture became more compact and abstracted heavily from end users.

Data monetisation also played a role in another important transformation of big data. Manufacturing, healthcare and governance sectors, for instance, relied heavily on data collected through an array of remote IoT devices. In order to improve speed and reduce the load of network, the need for localised processing resulted in the birth of what is called ‘edge analytics’. The sync between edge and centralised platforms for data exchange and analysis is symbolic of the high degree of fine-tuning now possible in big data implementations.

For successful big data adoption, a baselined big data strategy is the core ingredient. The strategy should outline basic principles of architecture, analytics needs and transformation roadmap. Instead of starting from scratch, enterprises can adopt industry-focussed solutions by bringing in assets and accelerators. The solutions can be customised and transformed piece by piece in alignment with the strategy. That is where enterprises can get time to enhance their in-house capabilities, establish an analytics centre of excellence (CoE) and governance frameworks to transform all aspects of people, process and technology.

Enterprises need to ensure that architecture components are agile and pluggable. It is better to take small steps with shorter implementation cycles rather than go for a ‘big-bang’ implementation. While designing solutions, architectures should be encapsulated with use cases rather than the other way round, to keep them more grounded and easy to evolve.

The alliance of big data with blockchain will potentially change the way we see big data for now. We are yet to fully converge the impact big data analytics will have on making automation systems intelligent. Intelligent chatbots, personal assistants and automated analysts are some glimpses of a big data-driven future.

Sudipta Ghosh is partner and leader, Mukesh Deshpande is executive director, technology consulting, and Pankaj Sharma is principal consultant, PwC India.

Close