A featured contribution from Leadership Perspectives: a curated forum reserved for leaders nominated by our subscribers and vetted by the Retail Tech Insights Advisory Board.

Ollie
Cheap to Build, Hard to Prove: What AI Changes About Customer Retention


Jonas O. Klink
Jonas O. Klink is a product management leader with extensive experience. He specializes in product strategy, user-centered design, machine learning-powered conversion optimization and building high-performing global teams that deliver business results through technical innovation and cross-functional execution.
Measuring What Lasts
At Thrive Market, our first AI project cut customer service costs by about a quarter, with no drop in CSAT or NPS. The automation took routine contacts off the queue, which left agents time for the conversations that needed judgment and a human touch. That was the easy kind of win.
Retention is a different story. AI has made it cheap to build experiences that might attract and even keep customers. It hasn't made it any easier to know whether they worked. I think that gap is where a lot of teams will get hurt.
At Thrive, we grew session-to-membership conversion 20 to 30 percent a year, compounding over several years, and we paired it with retention work so the gains held. I'm not against optimizing the transaction. But I have watched teams fall into a trap, and I see it at the intersection of AI and commerce today. Checkout conversion shows up in days. Retention takes months. So teams optimize what they can read by Friday, and ship wins that fade when you check the same customers a quarter later. In a subscription, the first order is the cheapest part of the relationship. The value is in the fifth.
A discount that lifts checkout looks like a winner until you check whether those customers place a second order at the same rate. App downloads work the same way: installed and logged in within a week is the number I manage teams to.
Where AI Helps Most
Retention has three levers I focus on: earning a longer commitment up front, saving customers who head toward cancellation at a reasonable cost, and getting people to actually use and enjoy the product. The third is the biggest.
At WW and Thrive, my teams ran a stage prediction and nudge program for it. We sorted customers into cohorts based on where they were with the product: struggling and close to canceling, building a relationship, achieving meaningful milestones, or thriving. AI predicted which cohort each person belonged to and was a thought partner for generating and testing the nudges meant to move them along.
“A discount that lifts checkout looks like a winner until you check whether those customers place a second order at the same rate.”
Every cohort also carried a predicted lifetime value (pLTV) score, so we could put a financial value on moving someone from a lower-performing cohort to a higher one. That's where the economics are, and the program grew the overall LTV of our subscriber base by double digits, measured against holdout groups of customers who didn't get the nudges. But a pLTV gap tells you what a move is worth, not whether you made it.
Interruptions have a cost, even the ones that look free. At one of the subscription businesses I worked at, we tested removing a pop-up that interrupted shoppers to collect emails for abandoned-cart campaigns. Visitor-to-paying-subscriber conversion rose 8.9 percent in relative terms, with no detectable change in churn. Now, picture an AI agent triggering thousands of interventions automatically. Some will save a customer. Some will go to people who would have stayed anyway, and some will annoy them. Telling those apart is the hard part.
The line to watch is helping versus trapping. A nudge that makes the product better for a struggling customer is retention. One that makes canceling harder is a risk you'll pay for later.
Why it’s Hard to Prove
Customers in the thriving cohort were different to begin with, and happy customers use features, so the gap between cohorts can't tell you what a nudge did. Only a holdout group tells you how many people you actually moved, which is why every nudge test we ran had one.
We have to design it before launch, because we can't rebuild it afterward. AI also makes it easy to ship ten nudges where you used to ship one, which means ten things to measure properly. Before you fund another AI build, fund the people and tooling to prove it worked.
What I’d do Monday
This applies to any retailer with a loyalty or membership program, whether subscription-based or not. Start by reviewing your last five winning tests and examining how those customers behave after 90 days. Choose one early retention signal and make it a guardrail for every test alongside conversion. Then define what building, achieving and thriving look like for your product and estimate the value of a customer at each stage.
Write down the decision rule before anyone sees the results. I apply the same discipline to every LLM pilot we are scoping at Ollie: identify a real customer problem and agree upfront on how success will be measured. An AI feature without a measurable outcome is just an expense with a good demo.


