The ROI Problem of AI
Understanding issues on the bottom line
“It’s not about ideas. It’s about making ideas happen.”
-Scott Belsky
AI currently has an ROI problem. It is clear that AI can create value but proving and realizing that value is less clear. That was my main takeaway from HumanX back in March (where I spoke about AI personalization). Most organizations still are not seeing a bottom-line impact from AI and stakeholders are starting to demand proof. AI is great for the individual, but it is a bit murkier for the organization.
If you think about a company, you can distill its effectiveness down to a simple ratio. Revenue over cost. If this ratio is greater than 1, you're making money. If it's less than 1, you're losing money.
When you think about the impact of AI, it gives you expanded capability and a lower time or cost of delivery. That is, you can potentially increase your revenue by providing increased capacity that wasn't possible previously or you can capture revenue that wasn't available previously. Alternatively, you could deliver the same output either faster or cheaper. The outputs from AI don't have to be exclusive to either bucket and we routinely see it falling into both. However, AI is not free. It incurs both usage costs and ongoing management and maintenance costs, all of which can be non-trivial.
Not many question AI's ability to do the above. They tend to question the risks, degree, and effectiveness around the implementation. What they really want to know is whether applying AI will hurt the business long-term and whether the promised value will actually materialize. This tension is occurring because a gap currently exists between the creation of value and the extraction of value.
This is a nuanced point that many are missing in the current applications of AI. Just because value can be easily created by AI does not mean you are able to extract the value easily. Value extraction comes mainly in two forms - increased capability/market capture and decreased headcount. The problem with the first is that all of your competitors might be doing the same thing, resulting in your edge diminishing. The problem with the second is that you can't always automate an individual's job entirely. Let's look at some examples.
Software Development
Imagine you're running a team of developers. You're likely seeing development gains of 30%, maybe 50%, maybe 10x. It really depends on what you're building and what type of development you're doing. But how do you realize those returns? You either need to increase the output in terms of features per unit time or overall capabilities of your organization. Alternatively you can cut your headcount and have the same output with less people. However, coding is not the only part of a developer's job. Part of the job is to assess what needs to be done, how to break up problems, and the best way proceed over the long term without disrupting what's already in place, along with might need to be built. There is a proactive nature to the job that requires knowing what questions to ask and what concepts to apply in a timely manner. So even if your developers can code 10x faster, it doesn't mean you cut down 90% of your developers. Additionally, there may be bottlenecks outside of the development pipeline the prevents you from fully realizing value. For instance, if your team needs to run in-market tests, you will need time, depending on the scale of your organization, to reach a sufficient number of observations to validate the tests of different features. This could be a week, two weeks, or three months. If you need to wait on that test to complete before moving to different features, it's unlikely you'll be able to extract the true value of accelerated development. This means the modes of operation likely need to change.
Quick-Service Restaurant
Alternatively, imagine you're running a quick-service restaurant. You've optimized heavily and only need five people to run it per shift: a manager to make sure things run smoothly, two cooks to make all the food, an in-store cashier to handle walk in orders, and a drive-through cashier to handle drive up orders. You've identified two places you could apply AI - robots for cooking and a voice AI to handle drive-through orders. Will you be able to realize the value of AI and cut your shift staff from five to two? Doubtful. For two reasons. The first is that the last mile of AI is difficult, and the second is that because you've optimized your store so well, each member of the staff is performing multiple roles. So while you may have an AI that can take orders well, the drive-through cashier is still taking payment transactions, filling drinks, and delivering the meal to the customer. Meaning you can't completely cut their role nor likely shift their tasks onto others. For the robotic cooking automation, the cooks still need to inspect the quality of the output, clean the station, refill ingredients, and generally handle edge cases. As the throughput of your restaurant rises due to additional capacity of automated cooking, resolving issues quickly becomes paramount. Minimizing time to fulfillment per customer is paramount to maximizing customer happiness and throughput at your restaurant. That might mean having a larger team to prevent issues.
Untangling
There are many situations where it is a lot easier to build from scratch, rather than to fix or make adjustments in place. This is part of why I think smaller organizations are thriving with the use of AI. They are decreasing headcount in a "phantom" way by never hiring the previously required individuals in the first place. This frankly makes the job easier as they don't have to figure out how to change existing processes to take advantage of AI while still maintaining or improving previous output. This is the current challenge in large organizations because they are much more complex and have hit many, many more low probability, impactful edge cases that resulted in large process changes. Think of security or fraud prevention at banks. There's typically a five step process or more to verify your identity to ensure that you are who you say you are and can make transfers. There are so many steps because the prior processes were exploited by malicious actors to make fraudulent transactions, at great expense to the banks.
As a large organization, you are embodied by many intertwined complex processes to make your business run. Part of this is by design and part of this has been built up over the years due to time and resource constraints. Without ripping up a lot of this tangled web and recreating how things are done, it can be difficult to fully realize the value of AI. The above examples were simpler and more group/team oriented. Shifting towards trying to automate a process that touches multiple parts of an organization is much more difficult, because the amount of ways something can fail grows exponentially.
Even if you could unknot every strand, another issue is still there. The issue of the Red Queen effect is often left out of ROI calculations. That is, in order to remain competitive and maintain, not just gain share, is your industry forced to use AI? Does AI usage become table stakes or does your new baseline require the use of AI? In that scenario, ROI should also be measured against market share you would otherwise lose. If you're struggling to extract the value you perceive from AI, here are some questions to help guide you:
Can we align incentives with our AI provider so that we only pay if true improvements are realized?
If we automate or improve this component of the process, what are the next bottlenecks that need to be resolved?
What other tasks do our resources need to accomplish besides the one we're automating?
How intertwined are our existing processes and what happens if the failure rate to one component changes? What if the capacity changes?
Are we trying to use this to increase revenue or decrease cost or both?
Where have we failed to extract value from a tool we love and what were the repercussions?
Good luck extracting.