Our last post discussed the confounding problem in present-state AI for marketing, and pointed out its ubiquity in the industry today. But how and why are companies still buying this technology? Read on.
Once we understand the fatal flaws inherent in predictive AI and contextual bandits, the first natural question is: If this is such a flawed technology, why is everyone using it, and why is everyone buying it? Great question!
There are several issues to unpack here.
First, the actual measurement and attribution of changes in lifecycle marketing campaigns is difficult to measure in and of itself. Most companies can't agree on how to measure it between each other, and often can't even agree on a standard measure from one department to another.
Second, in the absence of a good result, the technological failure is protected by an industry culture of opacity. The methodology of performance measurement is not defined in advertised results, so unverifiable claims are everywhere. Without objective benchmarking, vendors naturally present "case studies" showcasing huge uplifts but often without disclosing the full picture. This is really just a showcase of the various ways in which stats have been manipulated for decades in the pursuit of sales.
Cherry-Picking: Results pulled from a single, best-performing campaign, or limited time window, while underperforming ones are ignored.
Weak Baselines: "Uplift" measured against previous poor performing campaigns, including no communication, rather than the previous best-performing strategy.
Confusing Metrics: Success metrics with labels such as "improvement" which can be caused by factors unrelated to the technology itself. Classic example: The simple act of sending a higher message volumne, which creates a short term revenue bump but will often accelerate long term churn.
Bad Success Definition: Focus on vanity metrics (opens, clicks, downloads) while business-critical outcomes (profitability, LTV ) are ignored.
TL;DR - Without independent review, it's easy to claim anything. Why? Incentives!
Of course, most vendors cannot or will not provide validated average performance across their entire customer base. This information asymmetry means buyers are making multi-million dollar decisions based on selective evidence they cannot independently verify.
We think this is a big problem, and believe the industry needs to ultimately move towards an objective measure, and verifiability, but that is a very large topic.
For now, we're going to focus on what can be done about performance, and create the type of business results that will speak for themselves. Our next post will discuss the coming revolution, and why it may be something that most have not seen coming!