How often do data science projects fail? If we take popular internet media at face value, despite well-publicised successes, the graveyard of failed projects must well and truly be overflowing with corpses of those that “didn’t know better”.
Numerous articles exploring these failures touch broadly on the same raft of issues — lack of clear goals, too many silos, missing expertise, non-data driven cultures … sometimes blaming the business, other times faulting technology, but usually, artfully, indicting both.
The claimed failure rate, a cannily high enough number to grab the reader’s attention, ultimately ends up dividing the audience.
The cynic might say there’s a consultant or two skulking in the corner, waiting to pounce on the unsuspecting business!
What happens when things don’t work out
Budget pressures … sunk costs … disappointment … perhaps shame … and likely, business or career repercussions.
Through all of this though, what keeps popping up seemingly unnoticed is advice … and lots of it.
Search for “Why data science projects fail” and the resulting links are likely to point to articles with titles such as 7 steps to project success and 5 reasons analytics fail.
Two recent ones we came across are Doug Gray’s Data science and analytics failures and Ganes Kesari’s piece on Data science adoption challenges. They resonated because we’d experienced many of the points raised … and we thought they were nicely presented.
So you read the articles, the messages hammer home and you vow to make sure those weaknesses are addressed for next time. If you’re diligent, the more relevant points become a lessons learned checklist at the start of every project.
And eventually, you might start to believe that your formula, system, or solution, is robust enough to handle most eventualities.
Until reality bites
A customer some years ago was looking for data-driven analysis to identify potential savings with a key Managed Services supplier. Not coincidentally, it was close to contract renewal time.
Since the supplier worked in a Citrix-virtualised environment, with a bit of Splunk magic we were able to extract productivity data and visualise 10% worth of low hanging fruit, to be easily snipped off at renewal.
We were reasonably confident. After all the work had been requested by the customer and the data proved that the supplier was overstaffed.
So why not remove unwanted services at renewal? Or add new scope if worried about de-scoping the supplier?
Imagine our surprise when we received a polite pat on the back for good work and a “we’ll look at it in 6 months” nod.
The hidden handshake
It was several months later when one of our stakeholders alluded to what we could only call a “symbiotic procurement relationship”.
The apparent unproductivity was really margin for the supplier, who would have been extremely unhappy to see it go. In return, the business could call in favours as needed.
Ultimately, a few embedded sticky relationships meant that neither the most compelling of business cases nor the slickest of presentations would have altered business as usual.
An open can of questions
Was this something we could have anticipated in advance?
After all, the eventual challenge we faced did sit at #6 “Change is disruptive and not handled well” in Doug’s list. And also at #5 “Lack of executive buy-in” in Ganes’ — which we thought we had!
Could we have planned for it amongst everything else going on at the same time?
Is it humanly possible to be so intimately involved with every aspect of a project … so as to be in a position to pounce at the first sign of a top 10 problem?
Were we the only ones to feel that data science advice always makes perfect sense after the fact … but is overwhelming before the fact?
Eventually, grim economic reality forced us to be honest with ourselves.
Our previously airtight business case had neglected to consider tacit influence not because we didn’t want to … but because we didn’t have the relationships to know anyway.
It was a trigger to start looking at data science failures and successes in a different light.
And explore the question we’d been trying to avoid for some time.
In a very technical field like data science, how much do people, relationships and context matter … for success?
Afraid that the answer, perhaps unthinkably, might tell us it matters as much as data chops, critical thinking and A+ solutions.