Everywhere we look, data science — perhaps simply a combination of statistics, machine learning and analytics as Cassie Kozyrkov puts it in one of her articles— is on a hot streak. So much data generated, so many questions to ask.

What can we do with this data?

Where could it take us?

Will it improve our lives?

How can it change the world?

And let’s not forget…. can we make money out of it?

First, a disclaimer

It was a feeling of discomfort that started us reflecting on data science … not unlike the sensation left behind by a long-ago dream that one can’t quite recall.

As the years passed and we felt more angst after experiencing too many WTF moments, the dots appeared to connect more clearly.

They seemed to hint that while chasing expertise, career and business, we may have somehow been distracted … or misled … in our ideals and purpose.

And that data science, source of the so-called ‘sexiest job of the 21st century’, has a potent dark side that appears quite convenient to overlook while riding the bandwagon.

Caveat lector!

Where is everyone going?

Today, most mid-to-large organisations have dedicated analytics teams. Over time, we’ve been conditioned to think that if we understand our data intimately, we could use it to eke out more revenue or reduce a chunk of costs.

To encourage us, the world continues to invest huge amounts of time and capital into tools, algorithms and techniques. And then it teaches us to combine technology with human behaviour so we can zero in on what consumers really want…..and somehow shape their irrational buying decisions.

Solve bigger and better problems and humanity might just get to that holy grail of Artificial Intelligence — sentience.

Can data science give us all this?

From the outside, it’s hard to know where hype ends and reality begins. Inside the bubble, would we even stand a chance?

A productive science perhaps

In data science, as in all other fields, projects succeed and projects fail.

Pass … fail. Good … bad. On time … late. Money making … money losing. Seemingly always binary.

Search the web and it’s easy to conclude that the amount of waste must be significant. After all, tales of failures are everywhere — one 2017 Gartner study even claiming 85% as the project crash-and-burn rate!

Assuming these are credible numbers, one might shudder to imagine the number of precious hours and dollars lost forever to that economic wasteland of mis-productivity.

Unsurprisingly, the articles most referenced are also the ones quoting the most dramatic numbers. Coincidence?

Productive or not, at some point we started to admit that perhaps data science wasn’t as important as we thought it was. After all, it appeared to produce nothing — no food, no clothing, no shelter.

More likely, it could only influence production. To ideally be better, definitely faster, maybe cleaner, always higher.

But how often does it happen that way?

From within the bubble

Success or failure, we only have our own experiences to indulge in and pass judgment on.

And tempting as it is to promote our data science wins, what really brought us to our knees were the failures. Not necessarily the big or bad ones, but instead, the failures that occurred after a successful project.

It might have gone something like this — spending weeks on the business case begging for sign off, painstakingly cleaning (if you can call it) data, desperately hoping for a reliable model, head-spinning hours of analysis … and … finally … insights well and truly visualised, ready to be served with a side of solid numbers.

Something that had worked before.

And then, nothing…….. a project torpedoed by silence, sunk without a trace. Why?

So many questions left unanswered. Shaken confidence. And those insidious self-doubts start creeping in.

Where to next?

“Teach thy tongue to say I do not know, and thou shalt progress.” — the philosopher Maimonides (1135–1204)

Given data science’s potential to shape so much of our uncertain future, perhaps it is worthwhile looking for help with those oft-unasked questions, the ones without data-driven answers.

Dare we then look at this field through a different, much less transparent lens? One with many shades of grey? And explore the influence of the hidden hand…. with its subtle undertones and delicate nuances that can only come from humans at play.

To hopefully better understand why some things happen the way they do, how data science projects fail despite ticking all the boxes …..and others succeed when they possibly shouldn’t.

And maybe, just maybe, we might get a hint of how this field really behaves out there in the world.

(Sep 2019) Book Fair, Bangsar Village 2, Kuala Lumpur