About this project
Search & Discovery for Lip Products
Color Matching Beyond Brand Shade Names
Shade names like "Velvet Plum," "Midnight Berry," and "Spiced Rosewood" are evocative and aspirational, but not helpful when searching for a specific shade among thousands of options. Online shopping makes it worse: coarse color groupings like "Pink," "Berry," and "Nude" leave users scrolling through hundreds of products... and even those groupings span a wide range of shades!
So I built a color search and discovery tool that skips the names entirely, matching lip products by actual color using CIELAB and Delta E, the same standard cosmetics manufacturers use internally.
Behind the tool is an ML pipeline — image classification, semantic segmentation, and perceptual color clustering — applied to 34,000+ raw product images to build a searchable color index from scratch.
And it's a sizable market
$17.49B → $23.77B. The global lipstick market is projected to grow from 2024 to 2030 at 4.7% a year, an enormous category that still has considerable user pain points for search and discovery.
The why, what & how, in full below↓
WhyThe palette
How retailers organize the color spectrum (or don't)
Every major retailer collapses the entire color spectrum into about a dozen named groups — Sephora has 14, Ulta has 12. And while a dozen groupings might sound like a lot, a single category can hold hundreds of products and very different shades of the color group.
Take "Pink," for instance. The Sephora and Ulta palettes above already show that "Pink" means something different at each retailer. Apply that filter and the result is a wide range of shades, hundreds of products, endless scrolling, and no way to quickly find the color someone is searching for:
Marketplaces like Amazon and Google Shopping offer just a search bar and no color filter at all. Finding something more specific than "red" or "pink" requires fluency in makeup vocabulary: knowing that the shade in mind is "terracotta," "mauve," or "dusty rose." These terms mean nothing to a casual shopper. Moreover, discovery becomes impossible when there are no words for what you're looking for.
In fact, here is what searching "pink lipstick" returns:
The takeaway
Searching for "Pink" returns every shade below (and more) at once, making it difficult to discover and search lip products.
WhatThe approach
Search and discover by color
Color search works differently depending on how much is already known about the target shade. The three modes below cover everything from casual exploration to color-perfect precision.
- 1 For casual exploration, the color wheel groups shades by their distribution in the product catalog, with an option to zoom into lighter and deeper shades.
- 2 When a reference exists — a photo, a screenshot, something spotted on someone, or a product that needs a dupe — the photo upload identifies the closest matches within a perceptually imperceptible difference.
- 3 For full control, the hex input opens the entire color space: paste a known value directly, or open the color field and drag across every hue and saturation to land on exactly the right point.
Pick a shade from the color wheel.
Or sample a color from a photo.
Matching products ranked by similarity.
HowUnder the hood
Building the color index
Building the tool required an end-to-end ML pipeline, from raw scraped images to a searchable perceptual color index. Each stage is outlined below.
Scraped product listings and imagery across major retailers. Validated and repaired downloaded images. 12.5% of products shared the same stock photo and were removed before classification.
No labeled dataset existed, so one was built. The training set uses a multi-step strategy optimized for coverage, while a separate validation and test set is drawn for representativeness, so reported metrics reflect overall performance.
Trained a classifier to distinguish image types such as swatch, bullet lipstick, and liquid lipstick, among others. Image type determines the color extraction strategy, rather than applying a one-size-fits-all method. Images with no visible product color, like sealed packaging or multi-product shots, are flagged as unclassifiable and skip extraction: about 18% of all images.
Four segmentation models, one per image-type family, identify the color region — the bullet tip, the wand, lips on a model — isolating the actual product color from packaging and background. Each is scored by IoU (intersection over union), the overlap between the predicted mask and a hand-drawn one, where 1.0 is a perfect match.
Color is extracted in CIELAB space for each product by taking the median of the masked pixels, giving every shade a precise, perceptually-grounded coordinate. Images with no visible color are declined rather than guessed. A Gaussian Mixture Model (GMM) clusters over the full catalog to define color wheel.
Every product becomes a point in CIELAB space. So a query color returns the closest shades by ΔE, across brands, type of lip product, and price point.
Outcome
The web app lets users search products by color. For most products, the top match is as close to the target shade as the human eye can detect.
Pipeline code, model choices, evaluation, and the color-matching logic, are all documented on GitHub.
About me
Connie Schibber
Data Scientist who builds the measurement systems, experiments, and ML pipelines behind AI products: what to build, why they fail, how to improve them, and when to launch.
I'd love to hear if you found the app useful. You can connect on LinkedIn and include a message that you checked out the app!
Connect on LinkedIn