This article offers suggestions that can help you take your Machine Learning POCs a step further and develop a real operational business pilot. Often, we need to remind ourselves that our ultimate goal is not to train the best possible model but to harvest its potential value in the real world. We will provide here a few tips that are based on our experience and what has helped other clients to succeed. Some of these tips will be helpful before you begin, whereas others are intended for transitional or long-term use.
90% of the effort in successful machine learning is not about the algorithm or the model or the learning. It's about logistics.
Overview of a Machine Learning Project
It is common to split the development of machine learning system into three successive phases:
- Phase 1 consists of quantitatively measuring the relevance of the use case by sharing it with the business. This phase includes the identification and qualification of use case followed by an evaluation of business relevance and, if possible, ROI. If the use case is qualified, you start exploring the first machine learning models.
- The transition to Phase 2 consists of framing technically and functionally the use case. If the models are successful, you start developing a business pilot in collaboration with IT. This will include a variety of new tasks such as connecting to data sources, testing (unit, integration and performance), implementing a monitoring and audit system.
- The transition to Phase 3 consists of developing and deploying the pilot to collect users feedback and prepare for industrialization.
In this article, we will focus on the transition from Phase 1 to Phase 2 in Figure 1 and what kind of aspects you need to consider. Our goal is to give you practical pointers that can help make your ML systems success in production applications. If your ML model is successful in training and testing, there is no reason to stop here. It is time to develop a business pilot in collaboration with IT and end-users.
Don’t try to commit finalizing your model from the start (here we refer to the Phase 1) as you will know more a few months later about how your application fits into your IT landscape, who your users really are and how they want to use it. This is the purpose of the business pilot – explore the potential quickly during Phase 1 and validate technical and functional aspect during Phase 2.
Tip #1 - Connect as Quick as Possible to Real Data
The world changes and so will your data. When you put the ML system in production, there is almost always something you can’t control! Updated formats, new or corrupted data, loss of data sources will ask you to review your first model.
As soon as possible, you need to initiate collaboration with your IT department. It is becoming increasingly difficult to do serious development without access to production data. This will provide you with rapid feedback on whether this is needed to adjust in development and to involve other stakeholders.
Tip #2 - Prepare & Test it
The course of developing a new model is often highly iterative. This results in problems if the development history of a model is not well documented.
Being able to replicate past work exactly is incredibly essential, especially in the machine learning field. Otherwise, it becomes a real black box; nobody knows what is going on there.
Keep in mind that testing is not just a matter of unit or integration testing (even it remains important) — making sure that the various parts of a particular application will work together properly.
Tip #3 - Implement a Monitoring and Audit System
It is important to have close control over data and applications. As we mentioned previously, the world is changing – if your system working well in production today will need to be able to change effortlessly in future. And this requires knowing what is going on and being able to adapt to changing conditions.
Monitoring and audit system might also be really useful to detect the anomaly and abnormal behaviour in your ML applications automatically.
Tip #4 - Adopt DataOps Culture
Working with new technologies and old-fashioned thinking creates difficulties in production and tension in teams.
One way to avoid is to adopt DataOps culture. Already announced as the DevOps successor, DataOps is an organizational & technological approach to deliver Data projects. It aims to bring agility, automation and control between the different data project stakeholders, including IT (IT Ops managers, application developers, architects), Analytics teams (data product owner, data scientists/engineers, data stewards) and businesses.
Go further with developing business pilot will prepare you for future industrialization. The transition from Phase 2 to Phase 3 will be almost transparent: key stakeholders informed and involved, model fed with real data and testing ready. You might now focus on automation of processing, system monitoring and other continuous improvements.
