What Big Data Needs to Do to Grow Up

We are in an Information Revolution — and have been for a while now. But it is entering a new stage. The arrival of the Internet of Things or the Industrial Internet is generating previously unimaginable quantities of data to measure, analyze and act on. These new data sources promise to transform our lives as much in the 21st century as the early stages of the Information Revolution reshaped the latter part of the 20th century. But for that to happen, we need to get much better at handling all that data we’re producing and collecting. 

Consider the more than $44 billion projected by Gartner to be spent on big data in 2014. The vast majority of it — $37.4 billion — is going to IT services. Enterprise software only accounts for about a tenth. 

The disproportionate spending on services is a sign of immaturity in how we manage data. In his seminal essay, “Why Software is Eating the World,” Marc Andreessen pointed out that for each new technology wave, the money eventually shifts to software. Software spending represents an industrialization and packaging of work that would otherwise happen manually, as one-off services, within each organization. As markets mature, more of the processes move to partners and other providers, so the industry leaders can spend their time and energy on high-value processes that contribute to their competitive differentiation. Software is part of a broader ecosystem that lets businesses focus on activities that are core rather than context. Core, in Geoffrey Moore’s definition, is what a business’s customers cannot get from anyone else; context is all the other stuff a business needs to get done to fulfill its commitments. 

To understand what that path to maturity might look like for big data, it’s helpful to look at another, similar transformation. Data is the raw material that we attempt to turn into useful information. We can learn something from the manufacturers who turned raw materials into achievements as complex as automobiles. 

The earliest automobile manufacturers were “vertically integrated,” which is to say they pretty much did everything themselves. Contrast that with today’s automobile manufacturers, which source parts from a global marketplace of independent suppliers. A manufacturer like Ford might have more than a thousand Tier 1 suppliers. 

By calling on a rich ecosystem of industrialized products and services, automobile manufacturers can focus on the high-value, core activities that differentiate their products while driving down the total costs of production. This shift has led to a dramatic increase in automobile capabilities, without a corresponding increase in costs. 

Over the years, the automobile industry has embraced numerous innovations to achieve this transformation. These include: 

Standardization: From parts to specifications and protocols, standardization is the essential first step in building a mature ecosystem. 

Quality testing and controls: Standardization also includes the concept of quality controls, testing protocols, and acceptance testing to enforce adherence to standards. 

Design for manufacturing/design for assembly: Integrating manufacturing and/or assembly processes into product design has reduced manufacturing and assembly costs at the source. 

The shift from vertically integrated manufacturing (doing everything in-house) to integrated and collaborative design, manufacture, and assembly has enabled us to build larger, more advanced and complex goods than ever before. It has reshaped our communities and lives.

https://hbr.org/2014/06/what-big-data-needs-to-do-to-grow-up/