Assessing Projects with DFV
A framework for creating value
Happy New Year! As people think about the potential for the new year, they typically evaluate projects and opportunities. I've seen a lot of projects fail, particularly in the machine learning world, because they weren't set up for success from the beginning. Let's discuss a framework that has worked great for me when assessing new projects and opportunities. Then we'll dive into how this applies for machine learning projects. I believe this framework originally came from the design company IDEO. Since I've never heard a name for this framework, let's just call it the DFV framework.
Desirability - Do people actually have a need for your project?
Feasibility - Is it technically possible to create your project?
Viability - Is it economically sustainable to create your project?
Let's look at how to apply this in the case of machine learning projects.
Desirability
Do people actually have a need for your project?
Surprisingly or not, a lot of machine learning projects fail because there was never a true need for the model that was created. When we talk about desirability in the context of machine learning, we're really asking 'are we solving the right problem?' I've seen clients (both internal and external) ask for a myriad of things that never get to what is truly needed. They might be things such as:
We need you to build a neural network for this problem
We have to improve conversion rates by 30%
We need to cluster customers
We need to forecast our least profitable items
We need to predict our call center times
We need a random forest to predict market share
The wise reply to these statements is to ask why that model/metric/objective and dig deeper to the root problem and desired outcomes. Oftentimes people have not done the data work to understand where in the process there is an issue nor do they have an understanding of the places where machine learning can be most powerful. At the end of the day, most machine learning is really about automating a process. This means making sure you are solving the right problem in the right part of the process. You need to determine if the person making the request wants their request fulfilled or their problem solved. These are typically not the same and they usually want the latter. Find out what is desired instead of blindly accepting what is asked. Some questions that might help determine the actual problem are:
What will solving this problem help you achieve?
Why the focus on X?
What does this look like end to end in a perfect world? How is that different from now?
What changes if we implement X?
What impact are you hoping to achieve?
Feasibility
Is it technically possible to create your project?
This is the area that engineers and data scientists tend to spend the most time on, often while ignoring the other two pillars. I have found that many data scientists and engineers tend to focus on their task within the project. However, they must also work to understand what the future state of the project will look like after the model is finished and ready to be implemented. When we talk about feasibility for machine learning we are attempting to ascertain if we are modeling the problem correctly, the inputs are sufficient, the required performance is plausible, and if the process around the implementation will be sufficient. This requires asking questions such as:
Does the data exist? At the right frequency? At the right granularity?
Do you have the ability to serve the model or is other infrastructure needed?
How will you integrate with or change the current process?
How good does the model have to be?
How often are predictions made and at what scale?
There is an extensive amount of hands-on assessment that is typically needed for machine learning projects. In the words of Richard Hamming,
In science, if you know what you are doing, you should not be doing it.
In engineering, if you do not know what you are doing, you should not be doing it.
Unfortunately machine learning is a blend of both science and engineering. This often manifests in knowing which components and processes are needed but not knowing if the desired effect will be achieved until we get into the work. This comes as a surprise to many non-practitioners who think performance can be known without diving into the data and establishing baselines. Many times the data is not sufficient in either quality, frequency, or granularity and additional systems need to be setup to collect it. If the data does exist, a large portion of the allocated time is needed to clean, transform, and understand the data. From there, models can be constructed. Once a model is sufficient, there is still the engineering challenge of fitting the model into an existing process, creating the required infrastructure, and maintaining the model. All of these facets need to be probed to ensure a successful implementation.
Viability
Is it economically sustainable to create your project?
When creating a business or new product you want to make sure that enough revenue will be generated to sustain operations. However, when we look at machine learning projects what we really want to know is if we achieve the required performance, will we have a substantial positive impact on the business that is worth the required cost and effort to create the model? I've seen situations where a 1% increase in performance can generate hundreds of millions of dollars in additional revenue and other situations where a 20% increase in performance barely hits $100k. Each of these projects could seemingly take the same amount of effort. It's all relative to the leverage point in the business and solving the right problem. When you have scale, small increases can make a big difference. Alternatively, if you do have scale you can work on a problem that is not the chokepoint and has minimal impact on the business. Some guiding questions would be:
Will the engineering time be worth dramatically less than the benefit gained from this project?
Is there enough budget to successfully achieve the desired outcome?
How do we manage risk in this project and model?
Do we have enough time and budget to experiment correctly?
How much leverage will this model provide?
If this model was 100% correct (almost never happens), how would that affect the downstream process?
What is the impact of getting a prediction wrong? (This will happen and needs to be managed.)
Example
Let's walk through a real-life example and filter it through the framework.
ACME Corp
I once worked with a client that was trying to obtain 50,000 new signups in a single marketing campaign with a campaign budget of approximately $10,000. Our provided goal was to create targeting models to help improve the clickthrough rate (CTR) for this campaign. For this company, the average cost per thousand impressions (CPM) was about $12 and the average CTR was about 2%. Apparently, no one had done the back of the envelope math because based on those average rates, a campaign budget of at least $30,000 (triple the allocated budget) was needed, and that assumes a 100% conversion rate after clicking through. To obtain the goal we would have needed to generate a clickthrough rate of about 6% from our models. That was a seemingly impossible task. A CTR of 2% was already stellar for this sector and the need to triple it meant the project was not setup for success. The only available move was to work to set a realistic budget for the campaign. Let’s move through the DFV framework.
Desirability - Unlikely. We would want to know why the focus on 50,000 signups and why the focus on clickthrough rates? Knowing that a 2% CTR is very good, we would want to look at other parts of the marketing funnel such as awareness or conversion.
Feasibility - Not in the current context. Machine learning can do a lot of things but it is not magic. Machine learning is an accelerant. Achieving a 3x performance over the top human performance is extremely unlikely if focusing on a single piece.
Viability - No. The current budget allocation was insufficient for both the project to be a success and for the time it would take to successfully model the problem. You would spend more in engineering time to understand and clean the data than the entire campaign budget. The hope would have to be that the models created in this project could be extended to future campaigns.
As you enter the new year I hope the DFV framework helps you evaluate new projects and opportunities. To recap the three pillars are:
Desirability - Do people actually have a need for your project? Are we solving the right problem?
Feasibility - Is it technically possible to create your project? Is the required performance possible?
Viability - Is it economically sustainable to create your project? Will the impact be worth the cost?