According to Gartner, only 53% of machine learning models convert from prototype to production.
The statement above can also be articulated like this: 47% of machine learning projects fail to go into production. If an organization is set out to implement a machine learning solution in the first place, there can be a big chance that the implementation will fail.
Reasons for Failure
There are several factors that can lead to the success or failure of a project. Let’s look at some of the reasons and some solutions that could be used:
Expectations
When starting out a project, the business owners and stakeholders have high expectations for machine learning models. However, problems can arise if the business expectations and the machine learning model being built don’t align with the goals.
Machine learning is not magic. It works with the problem to be solved as a target and everything is built around it.If there is not a defined goal and a problem that needs to be solved, no amount of data analysis and modeling work can make the project a success.
Having a good understanding of the organization’s data maturity can help in identifying if the problem at hand actually needs a machine learning solution or can be solved by a simpler analytics solution.
Understanding that machine learning is an iterative process, it is important to have a success metric that defines when to stop. If there is no well-defined success criteria, the project will keep on going in loops.
Solution: Understand the Why
Before even deciding that the project requires a machine learning solution, it is important to listen to the customer’s problem. Understanding the why behind the customer’s ask for a machine learning solution can lead to multiple solutions.
It will help in defining the goal and create an understanding between the business and the data scientist, enabling them to be on the same page and work towards a feasible solution.
There are chances that the problem can be solved by an analytics solution and instead of a machine learning algorithm.When defining a business problem, the next part is also to work on defining the success criteria to make sure that the project is focused and has a fixed target to achieve.
Data
Provided there are clear expectations on the outcome from a machine learning project, the next failure reason can be the very starting point.
Machine learning models are as good as the data. For example, if we train a model to identify oranges from an image, and the data we have consists of photos of apples, the model will not know what an orange looks like. Such a model will not provide good results.f the problem statement is defined, there are factors that can impact the model training process, such as:
- Availability of the data required for the problem
- Data quality
- Relevance to the problem
- Data bias
Provided that we get the most relevant data to the problem, still, the majority of time is spent in data quality and cleansing work. The data required for training a machine learning model has steep quality requirements. If we are passing low-quality data during training it would result in a bad model, which would also result in inaccurate predictions. Moreover, even if the training process is streamlined to change the quality of the data, because of the way business works or the way the data capture processes work, the model could encounter bad data during the prediction process.The data we have for training is a representation of the data that the model will encounter when deployed in production.
Solution: Educate and Communicate
Not every customer is well-versed in data science terminology and the impact of data on the predictive model. Working with the customer to help them understand the overall process and how things work can benefit both sides.
It is important to incorporate data quality standards in the project and work with the customer to identify and eliminate the root cause of the bad data problem.
Model Generalization
What is a generalized model?Generalization, in simple terms, means that the model’s training is actually useful in the real world.
This can be a problem if the training data does not represent the actual production data. If you train a model on a strict dataset and it encounters a data point that it has never seen before, it will fail. There are two terms that we hear whenever we are talking to a data scientist: overfitting and underfitting.
Overfitting is when your model fits the training data too well and it fails to work with unseen data. One reason for this is using a method that is too complicated for a relatively simpler problem.
Underfitting is when your model is not able to identify any meaningful patterns in the data that you passed. One reason for this is using a method that is too simple for the problem at hand, or because you aren’t using the right data in your model.
Solution: Have a Good Model Training and Validation Process
Have a clear understanding of what “clean and representative” data means and work with the customer to get the same. Identify the right modeling technique to use per the data. A complex non-parametric method may work well in training but it could be possible that it’s not suited for a dataset that is relatively simpler.
Another approach is to use validation methods while training the data. There are various methods that can be leveraged, such as using resampling methods (cross validation) and following the train/test/validation set strategy where the validation set is used at the end of the training cycle and the training and testing datasets are used for training and tuning the model.
The “Proof of Concept”
A lot of times organizations would want to test the waters and see if the machine learning project is feasible. They may do this because they have doubts about the quality of their data, the value of machine learning, or limited budget to explore a major AI project. In this case, some organizations opt for a “proof of concept” (POC) project where there would be limited budget, limited time and, of course, limited data and resources.
With a project that has limitations, organizations try to validate if the project would work in a larger setting. The problem here is in the identification of the risks that we could encounter when implementing the project on a larger scale.
The training data that we used might be good enough for the POC, but when we look at the larger-scale data, multiple data sources may need to be joined, aggregated, or transformed. It could be possible that the data is not fit for the problem we wanted to solve or is not complete. Data science projects take time and include experimentation to identify what the best method is for the problem. Trying to replicate it in a smaller time frame and expecting high accuracy and insights from it increases the chances of failure, particularly if proper expectations are not set.
Solution: POCs are Not Bad, but the Execution and Expectations Are
It is important to work with the customer to identify and properly curate the data. If that’s not possible, the efforts should be spent in identifying any issues pertaining to the quality of the data.
Setting up realistic expectations with customers can go a long way. Before starting a POC, set the right expectations and evaluation criteria, as there is always room for improvement and the possibility of finding new insights when more data is incorporated.
Fitting a Model and Getting the Predictions
So we identified the machine learning requirements and their reasons, got the data from the customers, formulated the method we want to work with, and fitted a great model.Now what?
Failure comes when the utilization of the predictions is not clearly articulated and the users don’t know what to make of the output. It is important to develop methods to deliver the results in the right way to the end users and educate them on the meaning.
Solution: Begin with the End
Predictions from a machine learning model are just numbers if we don’t know how to use them. It’s important to identify how the model fits in the overall business process and how it should be used.
That said, engage with the functional users regarding their point of view, decide on the change management strategy with the stakeholders, and create a plan for end user training about the predictions and what they mean. At the end of the day, this step is just as important as the data and models.
Learn about Elevate and how we can help you succeed with your machine learning models.
