It’s really common to get hung up on optimising a particular metric, rather than thinking about improving the entire system.
I’ve usually seen this where we have a proxy for some desired behaviour that is usually good, but isn’t perfect. The classic e-commerce example is conversion at a particular point in the funnel. Increase this, you get more orders, which is a good thing!
However, some improvements to that number won’t work out overall. If you push more people down the funnel just to have a higher proportion drop out later on, then it’s not an overall benefit.
Even worse, sometimes that proxy metric goes backwards quickly, but it takes a long time to see the impact elsewhere and understand if you’ve got a better overall business.
In a big organisation, it can be tough to get people to look beyond the proxy. The day-to-day is focused on this single metric because that is the one they have scope to move. When the team makes a change, this number is one they have control over.
So, how can you step back a bit and make these overall system improvements?
First up, you need to make sure you are tracking your fundamental success metrics as well as the proxy. Order volume, value and profitability are some of the major ones depending on the stage of the business.
Next, be clear in your hypothesis. “We believe that by showing out of stock items sooner in the funnel, we will increase the number of completed orders, with a reduction in the conversion from basket to checkout”.
Finally, agree what your maximum decrease can look like. “If the reduction in conversion for customers seeing the new experience is more than 2%, then we will end the experiment early”. To get this agreement, you will need to work with all the relevant stakeholders, as this is going to cost money, and you need to agree that the budget to find out if the benefits really outweigh the cost.
Do all of this, and you’ll be able to make improvements to your overall system that negatively impact your imperfect proxy metrics.
Figure out “What’s the worst that can happen?” to step back and make the bigger change..