Only 15% of big data projects get deployed. How to get out of the crafting stage?
This astonishing figure was released by Gartner a few weeks ago. The first set of arguments to explain the phenomenon are technology-based: “Too often, pilots are built with ad-hoc technologies and infrastructure and are not created with production-level reliability in mind.”
TECHNOLOGY
When embarking on a data project you should ask yourself a few important questions:
- How to keep track of changing versions of analytical frameworks given the rapid pace of innovation.
- How do avoid technology obsolescence and gain access at the right time to emerging technologies that are not yet mature (at Saagie we dubbed this concept DTaaS – Data Technology as a Service)
- How to promote processing jobs and data pipelines between different environments, such as moving from a sandbox environment to a high availability production environment.
- How to integrate these processes within your existing IT environment.
- How to organize, secure and track the data in your lake (data gouvernance).
When you have appropriate answers to these questions, the chances to get your data project into production will increase significantly.
But you would be wrong to judge on technology alone, the reality is much more complex. Having the right platform is certainly a facilitator that allows you to minimize migration and administration costs, but the fundamental challenge is to satisfy business use cases, obtaining ROI and how to get your act together (team, methodology, onboarding…).
RETURN ON INVESTMENT and METHODOLOGY
Gartner puts forward a second set of arguments: “spending priority is low, which is caused by a ROI that is hard to prove or by the fact that big data experiments are only part of a larger funded initiative.”
Sure enough, measuring a ROI supposes you set tangible success criteria right from the start of the project. But delivering and proving a ROI is much more challenging.
A lot of factors contribute to success. Let me name a few:
- Think in terms of eco-systems (startups, suppliers, customers, competitors..)
- Work in a lean and agile way (test & learn)
- Find the right pluri-disciplinary team, internal and external.
- Take into account a wide variety of data sources including customer journey, IoT and IT systems
- Have an open mindset on business models
- Don’t forget the importance of onboarding and training
The list is obviously non-exhaustive. Let’s check this list again when the percentage of successful big data projects put into production hits the 80% mark!
One thing however is sure: it’s not going to be easy. It’s going to be a long and winding road.
Here is a prediction on what it will take in this challenging market to be successful:
- Ecosystem with the right mix of software and services
- A full-stack, end-to-end data platform that automates and industrializes
- The use of open technologies
