- AI Transformation
- Commerce
- A/B Validated
AI Search:
Recovering Lost Revenue
“Just type it out. We’ll find what you need.”
On Kurly, one typo could end a shopping trip. I led an AI search that catches what the old engine missed, so every search can lead to a cart.
- Period
- May 2024- August 2024
- Role
- AI Product ManagerA/B Test DesignCEO CommunicationQA
- Team
- AI PM (Me)1 ML Engineer1 Backend Engineer
- Org
- Kurly
Before
01 CONTEXT
Search Was Where Kurly Made the Most Money
Kurly is Korea’s premium fresh-grocery unicorn with 3.5M monthly active users. People who search already know what they want, so search drove more revenue than any other part of the app.
02 IMPACT
From Dead Ends to Discoveries
6.80% to 0.22%
No-Result Search Rate
58.60% to 60.14%
Search Click-Through Rate
$3.6M
New Monthly Revenue from Former Dead Ends
Featured at
Google Cloud Summit Seoul 2024
Google selected this project as a leading enterprise case of AI innovation on GCP. Kurly’s CEO presented it on stage, and I built the presentation.
Watch the talk →03 ROLE AND RESPONSIBILITY
A Three-Person Team, a Company-Wide Change
Users knew exactly what they wanted.Search told them it didn’t exist.
04 USER PROBLEM
Tiny Mistakes Left Users Empty-Handed
The old search engine only accepted exact matches, so a typo, an extra space, or a different word for the same thing meant an empty page.
Users were left with a frustrating search experience: guess the “right” spelling, try again, or simply leave.
What users want
Cabbage
Illustrative thought: Huh, no cabbage today?
Meanwhile, leadership believed the Bestsellers would catch them.
Curation is Kurly’s core value, and customers trust its picks. Even an empty search ended on a shelf of bestsellers, so it seemed like enough.
But I believed users who came with a goal would scroll right past it.
Based on data analysis, of the 6.8% of searches that found nothing,
only3.83%clicked a product in Bestsellers.
05 COMPANY’S PROBLEM
Small Fixes Kept the Big Fix Waiting
When a search found nothing or the wrong products, merchandisers and marketers asked the search engineers to fix it. Each fix was a manual keyword update, one request at a time.With the team buried in small fixes, the search engine rebuild, the work that mattered most, kept getting pushed back.
So I asked a different question. ↓
06 DEFINE
How might we turn “No Results” into results,
without rebuilding the search engine?
07 POC APPROACH
Teaching Search to Understand, Not Just Match
Prove it first, build later
First we had to prove that fixing No-Result searches was worth it, fast. So instead of building our own, we added an AI Search SaaS on top of the existing search.
RebuildAdd SaaS
Internal Demo: Same Typos, Products Found
In an internal demo, the PoC returned clearly better results for all four query types.
The quality was strong enough to ask for a full A/B test with real users.
What users want
Cabbage
Illustrative thought: There it is.
08 SEARCH FLOW
AI Search Only Steps In When Search Fails
I designed the flow to keep the search engine team’s work to a minimum. Every search still goes to the old engine first, and only an empty result calls AI Search.
09 A/B TEST
Would Users Actually Click on AI Results?
To prove the system’s value, I designed an A/B test around the KPIs we set in Define.
- Objective
- Improve Kurly’s search performance with AI Search
- Hypothesis
- Google AI Search will return meaningful results for “No Result” searches, lifting search conversion (CVR) and customer satisfaction
- Target
- All customers (3.5M MAU), 50% Control : 50% Treatment
- Duration
- 2 weeks
Test Results
ControlLegacy search
TreatmentAI Search
No-Result searches
Searches the old engine found nothing for
Click-Through Rate
3.83% to 26.50%
No-Result Rate
99.98% to 2.93%
All searches
Every search on Kurly
Click-Through Rate
58.60% to 60.14%
No-Result Rate
6.80% to 0.22%
Hypothesis Validated, Across Every KPI
I shared the test’s progress and results with every stakeholder, every day.With that result, I went to the CEO and got approval to integrate AI Search into the full system.
PoC
CEO Decision
Improvement
Integration
Hand-off
10 IMPROVEMENT
Safe to Scale, Easy to Watch
The A/B test proved the idea, but going live for every customer asked for more.
Before the rollout I improved two things: what reaches the AI, and how everyone can watch it.
1. A Proxy Server for Safety and Cost
Live traffic brought two new needs: production safety and cost efficiency.
I led the build of a proxy server between Kurly and Vertex AI that decides when the AI is worth calling.
Cache
Keeps results per keyword for 12 hours, so a repeat search doesn’t call the AI again.
Validate
Filters out abnormal and malicious queries before they reach the AI.
Block
Enforces the blocklist: unlisted brands, profanity, and medical terms.
Fall back
A circuit breaker shows the original screen if the AI is too slow.
As a result,
SaaS cost fell30.82%
with no change to key KPIs.
2. Monitoring for Performance and Demand
Slack Bot
Because failed searches are customers telling us what they want, I built a Slack bot that brings them to the team every morning.
It sends merchandisers and marketers the search queries losing the most clicks, so they can spot new trends and unmet demand fast.
AI Search Results BotAPP10:01 AM
2024-06-24 Market Google search terms · Top 30
CTR, highest to lowest
- 1. avacodo, 262 occurrences, 100%
- 2. strawbery, 179 occurrences, 80%
- 3. brocolli, 185 occurrences, 67%
- 4. qunioa, 82 occurrences, 60%
- 5. zuchinni, 174 occurrences, 50%
- 6. mozzarela, 218 occurrences, 45%
- 7. chikpeas, 160 occurrences, 41%
Illustrative data. Internal figures have been changed.
Illustrative thought: Why is everyone searching for this?
Dashboard
Because the Slack bot showed only the top queries, I designed and built a Looker dashboard for the full detail and daily performance tracking.
The CEO, engineers, and business teams all watched the same numbers in real time.
- Yesterday’s stats
- Daily trend
- Top search terms
Illustrative data. Internal figures have been changed.
11 INTEGRATION & IMPACT
Live in Two Months, $3.6M a Month
Two months after the PoC, AI Search was live for every Kurly customer.
Then I handed it to the Search Engine team, and it became part of their roadmap.
+$3.6M
New monthly revenue from searches that used to end in nothing
6.80% to 0.22%
No-Result Search Rate
58.60% to 60.14%
Search Click-Through Rate
35.52x
Return on AI Cost: Margin per Product vs. API Fee per Purchase
12 REFLECTION
What This Project Taught Me
- 01
Deciding with Data, at Scale
This was my first time making decisions on a large-scale A/B test with 3.5M monthly active users. I learned how to prove a hypothesis in numbers, use that evidence to align stakeholders, and keep improving the system on it.
- 02
Starting from the Problem
Instead of taking leadership’s existing belief for granted, I defined the user and business problems from data and led the direction of the fix. It was rewarding to see that turn into new revenue and a better experience for customers.
- 03
Building AI That Lasts
Bringing AI into a large, live service taught me that applying AI fast is not the goal. I learned to define clearly where AI adds value, and to build a sustainable system around it, with security, cost, and monitoring considered from the start.
