How to create a data-driven company3 min read . Updated: 22 Sep 2017, 06:59 AM IST
Data-driven companies use DataOps, a new way of managing data that promotes communication between, and integration of, formerly siloed data, teams and systems
Big Data analytics has become a buzzword and most organizations are rolling out initiatives to capture, store and analyse data to turn it into insights. This has led to the emergence of what is called a data-driven culture in organizations.
While earlier, lack of data-forced companies to take decisions based on intuition, abundant availability of data, by itself, has not helped in creating a data-driven culture. The big challenge for them: how to structure data captured from a variety of sources and provide access to the same to their employees.
Worldwide, it has been noticed that data-driven companies tend to infuse the culture of data-driven processes and decision-making through what is termed as DataOps—a new way of managing data that promotes communication between, and integration of, formerly siloed data, teams and systems.
In any data-driven company, access to data is of prime importance—and the culture of DataOps democratizes access to data. In traditional organizational models, information technology (IT) acts as a gatekeeper, deciding who can have access to which data. Here, gaining access to data could take weeks or even months, which meant that quick decision-making based on data took the back seat. DataOps, on the other hand, decouples the people who collect and prepare the data, those who analyse the data and those who put the findings from the analyses to good business use. With DataOps in place, people across departments have access to data and they can use data-driven insights to take quick decisions. When the DataOps culture is established, organizations will notice that data-driven initiatives are undertaken from the bottom of the company rather than just the top. This is exactly what happened in the case of successful companies such as Facebook, LinkedIn and eBay.
Besides the challenge of structuring data and making it accessible across the board, there are challenges related to skill sets. International Data Corporation predicts that by 2018, there will be need for 181,000 people with deep analytical skills, and a requirement five times that number for jobs related to data management and interpretation skills.
The exponential increase in wages for talented Big Data professionals is another challenge faced by companies. Managing huge amounts and variety of data, and then gathering insights out of that requires a large number of Big Data engineers and scientists—who are difficult to get and costly to keep. So, will this shortage of talent stunt the growth process? Or is there a way to circumvent this talent gap?
Organizations around the world are now looking towards newer technologies to reduce the heavy dependence on humans to resolve the inherent complexity in Big Data projects. Let us look at the data management platform itself. In simple terms, a data management platform is a piece of software that sucks up, sorts and houses information, and spits it out in a way that is useful for marketers, publishers and other businesses. However, there are a host of data engineers working in the background who keep the platform up and running. What if we can automate the data management platform? Well, that is the next big step towards reducing human interference—a data platform that manages and optimizes itself. While a fully functional, autonomous data platform is still years to come, data professionals are already working on automating certain aspects of such a platform.
Experts suggest that development of this automated platform will take place in three phases. In the first phase will be alerts, insights and recommendations: the data platform will flag alerts in case of anomalies, analyse how the platform is running in the background and will suggest ways to optimize it better.
The second phase will see “partial automation" wherein certain functions are intelligently automated, offloading any manual intervention. For example, engineers have to configure an underlying IT infrastructure—taking into account trade-offs between capacity, performance and cost. In the second phase of automation, engineers will merely specify at a high level what they need and the data platform will configure the infrastructure automatically and dynamically.
Phase three will comprise “fully autonomous behaviour" in which the functions will become fully automated: engineers won’t even have to specify what they need. The platform will look at past requests and predictively configure what will be needed.
With a fully automated model, the dependence on data professionals to service the data management platform will reduce to a large extent and they will be able to dedicate more time to convert data into insights. As the Big Data sector turns to a renewed approach to data management, organizations can get better in handling data with fewer Big Data professionals.
Joydeep Sen Sarma is co-founder and India head of Qubole Inc.