Big Data: inconspicuous, insidious, insightful and ubiquitous

Eventually, new technologies always get assimilated, no matter how shiny and disruptive they are.

They get taken up and integrated into definite ways of using, doing, and making stuff. At first, a disruptive technology shimmers with possibility. We can imagine this or that use for it. We can make this or that from it. We can do this or that with it. Genuinely disruptive technology stands out: It's conspicuous in a way that, say, older, established technologies aren't – and can't be.

Over time, the new and disruptive always become inconspicuous, however. And that's a good thing.

Believe it or not, big data is becoming inconspicuous. It's starting to disappear, to go away, as an obvious, even anomalous category. Some attribute this to a diminution of its hype, to dissatisfaction with its substance, to its many shortcomings in practice. And this is part of it. Big data has been over-hyped and, generally speaking, big data technologies do lack for substance. There's likewise no shortage of cases in which big data technologies have come up short – 'way' short – in practice.

It's no less true that companies are successfully using big data to do stuff, however. In a growing number of cases, too, people who are using or interacting with big data technologies aren't even aware that they're doing so. They just know they can do things they couldn't do before – like the retail analyst who can now query against 10, 15, or 20 years of historical data instead of one, two, or five. Or, WhereScape customers like Canadian National Railway (CN), which uses big data technologies to collect and store eight petabytes of telemetry and sensor data. This is data CN can use to proactively monitor its rail lines and rolling stock for maintenance, performance, and, most important – safety.

No configuration of conventional technologies permits use at this scale and at this cost. None.

In both of these cases, the people using and benefiting from big data technologies don't notice anything different.

Scratch that. They notice a big difference, they just aren't inconvenienced by it: nothing about their work changes – except their jobs get a little bit easier, or a little bit better.

The business analyst can suddenly use the tools she's familiar with to do stuff she couldn't do before. The railway company's maintenance supervisors, logistical analysts, or safety workers, are able to do their jobs more effectively, more intelligently – and much more safely. In these and other cases, big data isn't being forced or shoe-horned into highly specific use cases; it's being taken up and used naturally, organically, to address long-standing pain-points, frustrations, and needs.

Call to action

In a sense, this means the end of big data as we've known it: as a “thing” – albeit a mostly untried, mostly unproven shiny, new thing. If you're a data professional, this is nothing less than a call to action. If you can't relate to this experience – or, worse, if you don't have any inkling what's changing or why – you're somehow less of a data professional. It's as if you're a builder without a nail gun, a college-age kid without a Snapchat or Instagram account: a hipster without a boutique hobby or interest. In other words, you're in danger of becoming antiquated, outmoded – left behind.

You should be doing this. You should be hip to this. Your most imaginative peers are, after all.

And here is the important point. Right now, as I type, companies are moving from experimentation with big data technologies and big data architectures to real-world production usage. These aren't technology solutions in search of business problems. Yes, companies are using big data technologies to ingest and store data of different types, speeds, and sizes – a use case that just wasn't cost-effective using older technologies. But, just quietly, they are also using big data technologies to reduce complexity, eliminate costly software licenses (for commercial databases, for example), redundant server hardware, and other increasingly superfluous expenses. This usage doesn’t need masses of data, it needs an enquiring mind and a spirit of innovation – and we have no shortage of that in New Zealand.

The companies I've described aren't in any sense the New Norm. It isn't as if everybody, or even mostly everybody, is doing this. Forward-thinking, imaginative, innovative companies are doing this. Conscientious companies are doing this. These are companies that are cognisant of their responsibility to their shareholders, customers, employees, and (in the case of CN and others like it) to the average everyday people who could be impacted by service disruption or disaster.

These are the companies – and the data professionals – you should be watching and following.

Who knows: Maybe you can steal a march on them, too. 

Michael Whitehead is chief executive of data wharehousing company Wherescape