Ensuring Success in Modeling Projects

Introducing the Model Impact Thesis Framework

"The problem is not the problem. The problem is your attitude about the problem. Do you understand?" — Jack Sparrow

Not solving the right problem is one of the main reasons machine learning projects fail. This typically arises from a lack of stakeholder alignment. Stakeholders differ on which problem actually needs solving, or if they agree they have no idea of the true business value. Alternatively, if they know the business value, they don’t know what model performance is required to obtained that value. If stakeholders have agreed to that point, they still might disagree on project priorities, data availability, what’s unknown, or when to stop a project early. Doing this work upfront would prevent a majority of machine learning projects from failing by having a course charted towards success from the beginning.

To help you set up projects for success from the start, I'd like to introduce a framework and tool called the Model Impact Thesis. Just like an investment thesis states why we should invest our money into an opportunity along with the associated return, we want to lay out the logic for why pursuing a modeling project is a good investment of valuable developer time along with the associated return to the business. The basic idea is that before embarking on a modeling project, your team should align on the business goals the model is supposed to be solving along with determining what type of model performance is necessary to accomplish the business objectives. This simple process is about determining targets and mitigating risk. The various components that need to be answered are as follows and should fit concisely on one to two pages so they can be quickly understood. Value clarity and direction over verboseness and density.

• Business problem to be solved

• What is the value of solving this problem

• Model problem to be solved

• Project priorities

• Current industry benchmarks for this problem

• Current business benchmarks for this problem

• Target performance

• Performance value gradations

• Data availability

• What don’t we know/What do we need to research

• Other considerations

Details

A key part of doing machine learning well is risk management. When these systems are implemented they automate decision making, meaning unplanned interventions become harder. Therefore, we need to be clear on goals, targets, benchmarks, and expectations before making expensive time investments in model development and deployment. Here’s what we need to answer for each component of the Model Impact Thesis:

• Business problem to be solved - This is a description of what we are trying to solve through the creation of the model. This question alone can create a lot of debate and digging in order to get to the root problem that needs to be solved.

• What is the value of solving this problem - This is typically, but not always, a dollar value that comes from solving the problem in question. It is important to quantify the opportunity in order to ensure that the time spent modeling is worth it. This question should also answer, what is the minimum required value to be shown.

• Model problem to be solved - The model problem we are solving is typically not the same as the business problem we are solving. We need to transform the business problem into a mathematical problem based on data we have available and actions we can take to influence the behavior.

• Project priorities - What’s important in terms of how we solve this problem? Do we care more about the speed of finding the solution or maximizing the overall performance? Does the model need to be interpretable or is it ok to be a black box? Do we care about how far the solution scales?

• Current business benchmarks for this problem - How good is our current process? Too many times I've been through a modeling process where a company does not understand the state of their current process. The mere act of figuring out how to automate the process results in the company realizing their process was not as good, nor as airtight as they initially believed. By knowing the performance of the current process you have an initial baseline to compare against.

• Current industry benchmarks for this problem - How good is the current process for average companies in the industry? Alternatively, if models are implemented, what does their performance typically look like? This will be very problem dependent. For some problems, 70% accuracy may be fantastic, and for others, 95% accuracy is terrible.

• Target performance - The target performance should be calculated based on what is required to achieve the minimum required value for solving the business problem. This may be back of the envelope math or it may be a highly involved computation. The idea here is to set a performance target for the model that if achieved can provide the necessary return for the modeling work performed.

• Performance value gradations - This is a more in-depth look at the target performance above. Here, you should have an understanding of the increase in value for every percentage point increase in performance over benchmarks or targets. This relationship may or may not be linear. However, in an ideal world gradations help you track model performance to real world performance as the system progresses.

• Data availability - In an ideal world, what data do you want? What data do you need? What data do you actually have access to? What data can you easily get? What is expensive but potentially worthwhile?

• What don’t we know/What do we need to research - This is about understanding the uncertainty around the project: there may be things about the process that are not known, the state of technology may be in flux, or the team may need to do some research to understand how difficult the task actually is. Use this question to set expectations up-front as to what may delay the project or reduce the ability to meet the required performance.

• Other considerations - This is a catch-all for other things we may need to consider in the modeling process. Is the time to compute model inference important? Will the results need to be contorted to fit correctly into a UI? Will we have scaling issues? Will our cycle time be extended because we need sign-off from the business on every iteration?

Usage

How should you use the Model Impact Thesis framework? Before you give the go-ahead on any modeling project, the business and machine learning teams should come together to fill out the Model Impact Thesis framework. This forces the teams to align on the business problem that needs to be solved, the machine learning problem to be solved (which is typically not identical to the business problem), the value of the solutions, and various targets and considerations. The outcome gives you an at-a-glance understanding of if you should make the investment in the project and at what point the project becomes too costly compared to the benefit.

After doing the work to set this up, a few things should be clear.

1. What business value you need the model to create.

2. What model performance is needed to achieve that business value.

3. How much of a gap you need to close with the new model based on your current system or industry benchmarks.

4. You can roughly equate every percentage point of increased performance into a dollar/business value.

5. An understanding of your gaps in knowledge and data.

Having the Model Impact Thesis completed now allows you to do a few other things that may be nonobvious:

1. You can create a baseline model and estimate how much effort it might be to achieve the target performance.

2. You can quantify the worth of a new data set based on how much the new data improves performance of a model.

3. As the project goes on, you will likely hit unexpected roadblocks or achieving a certain level of performance may become harder than anticipated. Since you've quantified the business impact, you have the ability to determine if the project should be cut. How? If you have quantified the business value correctly, then you can compare the cost of engineering and machine learning time to the business value being created. If the cost to achieve the required performance will exceed the business value provided by the model, then it likely makes sense to cut your losses and end the project early.

4. You can check if the model metric you chose correctly moves the required business value.

5. You should have a checklist of things to quickly validate if your thesis for the model achieving the required performance is correct.

Examples

Now let's dive into a few examples to see how the Model Impact Thesis works in practice. We’ll walk through three examples:

1. Customer churn

2. Marketing optimization

3. Predictive maintenance

These examples will likely be a bit more lengthy than a typical Model Impact Thesis in order to provide background on some of the thinking. Normally you would fill these out to be clear, concise, and provide information at a glance. However, you would want to have backup calculations and logic to explain the outcome of each section.