- AI Transformation
- Commerce
- A/B Validated
AI Curation:
Scaling Themed Campaigns
“You name the theme. AI does the rest.”
At Kurly, every themed campaign on the homepage was picked by hand. When the team behind them shrank, the homepage went stale. I led an AI system that turns a theme into a full campaign and keeps it fresh on its own.
- Period
- June 2024– December 2024
- Role
- AI Product ManagerUX ResearchA/B Test DesignCEO CommunicationQA
- Team
- AI PM (Me)1 ML Engineer1 Backend Engineer
- Org
- Kurly
Campaign Theme Title
01 CONTEXT
Curation Is What Kurly Sells
Kurly built its brand on picks: customers trust that if Kurly shows a product, it’s worth buying.
On the homepage, themed campaigns bring a new theme and a fresh mix of products every day. It’s where customers discover something new and enjoy shopping.
02 IMPACT
From Hand-Picked to Self-Running
Implemented an AI system that curates as well as Kurly’s expert MDs. In an A/B test, its campaigns matched their hand-picked ones on clicks and purchases. With curation handed to the AI, MDs could focus on the work that needs their judgment most.
25%
of Key Homepage Slots Fully Automated by AI
200+
Themed Campaigns Run by AI
2days / week
of MD Time Saved
03 ROLE AND RESPONSIBILITY
Turning an MD’s Craft into a System
Customers came to Kurly
to discover something new 👀The homepage kept showing them
the same things 🙁
04 USER PROBLEM
The Homepage Stopped Surprising Anyone
Kurly customers love the products its MDs pick for them, and “the joy of discovery” is one of the company’s goals.
But as campaigns went untouched, the homepage lost its variety: the same campaigns ran for weeks on end, and with no one managing the products, more and more of them were sold out or no longer sold.
As a result, customers were losing the joy of finding new products.
Add to Cart
Illustrative thought: Let’s see what’s new.
The Sold Out and No Longer Sold labels are added to explain; the app doesn’t show them.
05 COMPANY’S PROBLEM
Fewer Hands, the Same 40K+ Products
Left on hold
Every campaign needed daily care as products sold out and sales shifted. With fewer MDs, that care stopped, and outdated campaigns meant missed sales.
“We know it needs to be done.It’s just been on hold for a while.We’re blocked.”
Illustrative thought: Don’t forget me!
🎄 Aesthetic Holiday Prep is On ›
But thoughtful human curation was Kurly’s core value.
So everyone was nervous about automating curation:
Could AI pick like a Kurly MD,
without costing customers’ trust?
So to learn how the experts curate,
I sat down with the MDs who do it every day. ↓
06 EMPLOYEE INTERVIEW
Where Does an MD’s Week Go?
To find the steps that took the most time and could be handed to AI, and to learn what MDs look for when they pick products, I interviewed 8 MDs and marketers, from juniors to seniors, and mapped every step of building a campaign.
1 · Steps AI could take over
Building one campaign took nine steps.The first five took the most time, and five of the nine could be automated.
- 1Trend Search
- 2Concept Definition
- 3Product Discovery
- 4Product Selection
- 5Enough SKUs?
- 6Campaign Proposal
- 7Campaign Approval
- 8System Registration
- 9Campaign Operation
2 · What MDs check before adding a product
Fitting the theme was only the first test.MDs also checked each product’s brand, recent sales, stock, discount, final price, and whether it was already in another campaign. That data sat in separate tools, so they looked it up one product at a time.
07 DEFINE
How might we let an MD manage a campaign with just a theme, without losing hand‑picked quality?
08 INTERNAL COPILOT
A Copilot That Cuts Curation Time
I designed and delivered Campaign Copilot, an internal tool that supports how MDs plan campaigns. Since only employees would use it, a custom front end wasn’t worth the time, so I chose Gradio to ship it fast and clean. It had two goals:
- 1. Cut the time spent finding and picking products. It reads what a theme is after, finds products that fit, and shows what MDs need to choose them in one view.
- 2. Raise curation quality until MDs are satisfied with it. Every search they ran and rated showed us where the AI fell short.
- Open to every employee
- 6 updates in 8 weeks from MD feedback
- Reads product descriptions, not just names
1. Type a theme. The theme subject is all it needs.
gradioResult
| Product | Price | List price | Discount | Discount rate | Weekly sales | Deal GMV | Content GMV | Stock | In campaign | Supplier | Release date | Master code | Deal code | Content code |
|---|
A rebuild of the real Copilot. Illustrative data. Internal figures have been changed.
How it works
From nothing but a short theme subject, the algorithm finds the products that fit.
Three steps do most of the work:
- 1. Refining the request into what the theme is after
- 2. Generating candidate products and keywords over several rounds
- 3. Ranking what comes back
- Theme input
- Refine the request
- Generate 10 candidate products and keywords× k
- Expand to similar products
- Rank
- Filter
- Final product list
6 updates to make it MD-ready
MDs’ ratings, feedback, and follow-up interviews drove every update. I announced each one and trained MDs on how to use it.
- 1.1
- 1.2
- 2.0
- 2.1
- 2.2
- 2.3
Features
New features that made MDs’ work easier and the results more satisfying.
- 1.2CSV ready for the CMSDownloads match the CMS upload format, so a list goes straight in.
- 2.0Scope and strictness slidersMDs choose how widely to look and how strictly to judge.
- 2.1Include and exclude keywordsMust-have and must-not words, which also made brand search work.
Performance
Not just features: searches got faster, more accurate, and cheaper too.
| Metric | First version | Final version |
|---|---|---|
| Search speed | ~1 minute | ~15 seconds |
| Minimum conditions met | Often missed | 87%+ |
| Cost per query | ~$0.50 | under $0.07 |
09 INTERNAL EVALUATION
Earning the MDs’ Trust, One Test at a Time
Before asking customers to see AI picks, I needed proof that the people who know curation best trusted them. I ran two checks.
1 · Which AI? A blind test
To choose the model, I tested whether our own algorithm, built around how MDs curate, beats a general-purpose AI, Google Cloud’s Vertex AI agent.
- 56 people, including the CEO, MD lead, and marketing lead
- 10 themes
- Blind: only Model A and Model B
Model A · Our algorithm
Model B · Vertex AI agent
![[Osulloc] Strawberry Green Tea Cake](/images/ai-curation/products/kurly-5116361.webp)
![[Fritz] Seal Cupping Mug](/images/ai-curation/products/kurly-5065874.webp)
![[Coffee Bean] Choco Mousse Cake](/images/ai-curation/products/kurly-1000052839.webp)
![[Blue Bottle] Mug](/images/ai-curation/products/kurly-5040751.webp)
![[Bread & Co] Earl Grey Cake](/images/ai-curation/products/kurly-1000093702.webp)
![[Teavine] Low-Sugar Home Café Syrup](/images/ai-curation/products/kurly-1000490315.webp)
![[Home Moment] Café Mugs](/images/ai-curation/products/kurly-1000370038.webp)
Our algorithm won 68% of 56 votes.
On plain themes the two were close. On themes that needed reasoning, ours won 77%: keyword search can’t tell what a theme leaves out or puts first.
A’s picks are from the test. B is illustrative: Kurly’s keyword search for the same theme.
2 · Would experts ship it?
A homepage-wide A/B test was a big call, so first I ran one last subjective review with the leads.
- A PM, the MD lead, and the marketing lead
- 21 test campaigns
- Scored 0 to 10
- Average score, out of 10
- 8.95
- Without the one theme that returned no products
- 9.4
Illustrative distribution. Average is actual.
The algorithm held up, and the MDs trusted it.
Now it had to work for customers. ↓
10 A/B TEST
Could Customers Tell the Difference?
To decide whether AI could run the homepage, I designed a 2-week A/B test around one question: would customers notice who did the picking?
- Objective
- Validate AI curation quality before the final integration decision
- Hypothesis
- AI-curated campaigns perform as well as MD-curated ones
- Method
- Same homepage slot, same theme; only who picked the products changed
- Target
- 24,278 users in A (MD) vs 24,455 in B (AI)
- Conditions
- 20 slots, at least 30 products per campaign, a new theme every day
- Duration
- 2 weeks
Promotional BannersLimited-Time OffersBest SellersSpecial CampaignsThemed Campaigns20 test slotsWhere it ran
The test ran in the 20 themed-campaign slots at the bottom of Kurly’s homepage. Both groups saw the same slots and the same themes.
Only the products changed: group A got the MD’s picks, group B got the AI’s.
125 themed campaigns in two weeks
A new theme every day.
Group A saw each one as an MD picked it, group B as the AI did.
A · ControlPicked by an MD
B · TestPicked by AI
KPIs are indexed to the MD-curated group (= 100).
Secondary KPIs
Hypothesis Validated, Across Every KPI
The test showed no difference between campaigns MDs curated by hand and the AI’s.In other words, MDs could now just set the theme and let AI do the curating.With that result, I went to the CEO and got approval to let AI run the themed-campaign slots.
Internal Copilot
Internal Evaluation
A/B Test
CEO Decision
Integration
11 INTEGRATION
Name a Theme, Get a Live Campaign
Going live had two sides:
a launch MDs could run on their own,
and the rules that keep every campaign full and fresh once it’s live.
1 · Launch from the sheet MDs already use
To keep MDs from having to learn a new tool, I kept the launch in the Google Sheets they already use. They add one row, and the AI fills the campaign and puts it on the homepage.
- 1Open the sheet
- 2Add a row: title, subtitle, start and end dates
- 3Done 🎉
🔥 Spicy Chinese Takeout Vibe ›
🍫 The Ultimate Snack Drop ›
2 · The rules behind the sheet
To fit the AI system into Kurly’s platform without putting the live homepage at risk, I set every rule it runs on, so each campaign stays full, fresh, and stable automatically.
Campaign sheet
Read every night
New and edited rows go in at midnight, and every live product list is rebuilt weekly as stock changes.
AI builds the list
At least 20 products
A thin campaign looks broken, so it isn’t shown. Products with mostly bad reviews in the past 2 weeks are left out.
Kurly’s CMS
Registered on its own
The AI is linked to Kurly’s content management tool, so each campaign is registered and kept up to date with no manual entry.
Homepage
A fresh lineup every day
Campaigns share the homepage’s theme slots, and each day the slots show a new mix, led by what customers click.
Miss the click target, and it comes down
MDs track each campaign’s numbers in the same sheet, and a weak one leaves the homepage.
12 RESULTS
Trusted With Kurly’s Homepage
Despite Kurly’s doubts about automating curation, AI Curation now runs 25% of key homepage campaign slots fully on its own. Proven on the homepage, it was rolled out to other parts of Kurly, such as Beauty Kurly.
On par
with hand-picked campaigns, with more diverse categories
25%
of key homepage campaign slots run fully by AI
200+
campaigns run by AI
2days / week
of MD time saved
13 REFLECTION
What This Project Taught Me
- 01
Defining the Right Bar
The goal was not to beat our MDs but to match them. Setting parity as success made a sensitive change, letting AI pick for a curation brand, something the company could say yes to.
- 02
Trust Before Scale
Putting AI in MDs’ hands first turned the people with the most at stake into the people shaping it. Their feedback made the model better, and their trust made the rollout possible.
- 03
One Step in a Bigger Plan
Framing this as Project 1 of a personalization roadmap kept every decision pointed somewhere. The Sheet, the hourly refresh, and the data it collects all set up what comes next.
