NEW YORK, United States —This past fall at an event in the penthouse of the Gramercy Park Hotel, I drifted past a YouCam Makeup kiosk. Set up on a giant iPad, it looked like a life-sized Instagram. I’d previously regarded apps of this ilk as social media-age video games for selfie-obsessed teenagers who like to see themselves with puppy ears, Disney-princess eyes and surrounded by a halo of twinkling stars — which is great (live it up, I say!), just so not me. Since a friend was manning the station (don’t want to be rude!), and in the spirit of no longer being an out of touch stick in the virtual mud, I approached the screen and selected a look.
Instantly, there I was in the most flattering lighting possible with Bordeaux lips, golden lids and tight-lined eyes — not so far from my usual, but with a little more lash (a fluttery improvement). The peach-bronze blush I was suddenly wearing was on the opposite spectrum from my go-to cherry flush and illuminated my face like soft-focus candlelight. My skin looked perfect: even, glowy, touchably soft — I would go so far as to say angelic. Even my hair (which I’d had coloured two days prior) looked amazing—the pale, perfect blonde of my dreams. Was that the app or actually me? Some cyber-entity had also beamed in and magically filled out my brows just a touch. Then — because it was addictive, and the results instantaneous — I tried a classic smoky eye featuring gradations of greys and browns (boring, pass), a 1980s Boca Raton aqua liner moment (hideous, hilarious) and, also for kicks, a thick wing of fuchsia liner — one deft swipe across bare lids — again accompanied by that ultra (yet tasteful) lash. I took a screenshot of it. I Instagrammed it.
It was graphic yet minimal; sophisticated and polished, but built for impact. It looked like something Peter Phillips would do for a Raf Simons runway. It struck me as the sort of simple look that could transform whatever I was wearing— how good would this be with a chunky grey turtleneck? — and was easy enough that I could actually pull it off myself. I came home and rooted through my collection of Make Up For Ever and Urban Decay eyeliners for something in the shocking pink family. Incidentally, I’ve also been thinking of changing my blush (after decades) and going warm instead of cool — a watershed moment, for me, at least.
Trying on cosmetics is a difficult thing to do in a retail environment, or on mobile or e-commerce. You’re taking a financial risk and not really sure what you’re getting.
Augmented reality, which started out as a novelty, as entertainment, as a first cousin of silly-fun Snapchat filters, looks to be turning into a powerful and hugely significant sales tool for the cosmetics industry. Wiggling its way into the beauty sphere via the seductive, instantly satisfying domain of magic mirrors, brands are betting heavily on it.
YouCam, which, in addition to its Makeup app, also offers stand-alone Perfect (for selfie photo editing), Fun (cat whiskers, anyone?) and Nails iterations, is working with “every major beauty brand out there,” according to Adam Gam, vice president of marketing at parent company Perfect Corp. They’ve signed global contracts with L’Oréal and the Estée Lauder Companies, as well as deals with department stores like Macy’s. ModiFace, the industry leader in AR-assisted shopping founded ten years ago by Parham Aarabi, a professor of electrical and computer engineering at the University of Toronto, counts LVMH, Lauder, L’Oréal, Coty and Avon among the clientele for which it builds virtual “try-on” technology. In fact, “it might be easier to list who we haven’t worked with yet,” notes Aarabi.
The premise behind so-called "magic mirrors" is simple: They show consumers — who may either just be playing around or genuinely looking for direction — what they will actually look like wearing a specific shade, application technique, or complete look. Magic mirrors take the mental math out of studying a look on a model in an advertisement, an influencer on social media or an actress on a red carpet to imagining it on one’s own face.
Developers like YouCam have been using AR “to unlock a real pain-point,” says Gam. “Trying on cosmetics is a difficult thing to do in a retail environment, or on mobile or e-commerce. You’re taking a financial risk and not really sure what you’re getting.” Indeed, if you’re looking for a red lipstick—and there is an ocean of difference between brick and cherry, even cherry and berry — a magic mirror, on your phone, or at an in-store console, makes it relatively easy to cut through the clutter of, say, one hundred reds, or even 15 reds, to find that red. The right red. Your red.
Of course not all AR applications are created equally, and there are several critical keys to success.
1. It must be realistic.
“You need AR with really good facial recognition,” says Gam. Though YouCam has only been in business two-and-a-half years, it is a spinoff of a Taipei-based company, Cyberlink, which specialises in facial recognition and has the patents to prove it. “We had a lot of those patents going in, and that gave us a jumpstart,” says Gam. YouCam’s recently launched 3D engine enables live motion, “so you can turn all the way to the left, to the right, and the makeup maps your face. It doesn’t fall off like a filter might.”
Aarabi agrees: “People know it’s a virtual simulation, but they want to see something that looks realistic. Lipstick can’t be hovering somewhere near your lips; colour and placement must be precise,” he says. “There are Snapchat filters that are fun, but when someone tries these applications, the intent is to see if that shade looks good on them — and if the answer is yes, they end up buying that product.”
2. Colour and shade must be accurate.
Sounds basic, but this is actually pretty tricky: The colour in the tube/bottle/pan must look precisely like itself on-screen, instead of an approximate swatch. “We spent a lot of time validating each shade to make sure they are truly representative of the end product,” Kristy Frivold, senior director of Sephora’s Innovation Lab, says of the mind-bending 8,000 shades across 90 brands the company painstakingly inputted into their (ModiFace-powered) Virtual Artist, first launched in 2016.
And then there’s texture and transparency. “If a shade is incredibly matte, it should come across as opaque, as opposed to sheer or glossy,” says Frivold. “It’s definitely a science and it’s not perfect,” she admits.
Now, throw in to the mix the fact that the same colour will likely look different on you than on me. “That’s the other thing that’s important: making sure the underlying skin tone of the user is taken into consideration,” says Frivold, who says that Sephora’s software adjusts pigments depending on a user’s complexion. “The accuracy and realistic nature is what drives it from being a game into a real solution for beauty brands,” she says. “It’s SAS: software as service.”
3. It must be fast.
“The instantaneousness of the experience is one of the biggest factors for it to succeed,” says Aarabi. “People don’t want to wait for a result. Online, if you can do something in five seconds, you will see conversion rate and time spent go up. If they have to wait, the benefit dramatically drops off.”
It is for this reason that Estée Lauder integrated it’s (ModiFace-designed) Virtual Try-On function directly into the product detail pages on its e-commerce site, instead of giving customers something to download, or directing them to another page.
Once you’ve got all of that down, let the fun (and, hopefully, the sales) begin.
While ModiFace operates like a private label Oz behind the curtain — quietly developing and licensing the AR technology that powers many of the leading cosmetics brands’ and retailers’ try-on features — YouCam, which performs as a series of apps, is out front building not only its own brand, but its own community.
Billing itself as “a destination for beauty lovers,” YouCam boasts 525 million global downloads across the United States, United Kingdom, Europe, China, and Japan. They collaborate with retailers and brands, as well as makeup artists, celebrities, and beauty influencers to “bring their looks to life,” says Gam. They help Macy’s customers build complete looks, cross-shopping a range of brands “the way a real user shops.” With brands like Lancôme, which worked with YouCam to translate the seasonal colour story from a recent campaign, offering those looks to its base of users is a paid opportunity. With well-known makeup artists—like Wendy Rowe, Mélanie Inglessis, Sir John, and Robin Black, with whom they have collaborated — it is often a value exchange: “We give them exposure and they give us exposure,” says Gam. “They’re able to get their look in front of half a billion users, as if published in a magazine. They get hundreds of thousands of try-ons and millions and millions of impressions. We also produce editorials, to tell their story.” Depending on the partnership, sometimes specific products for purchase are coded into the look, sometimes not.
Like a beauty news website, YouCam reacts to and creates content around what’s going on in popular culture. Users can try on the main characters’ looks from the just-released film Pitch Perfect 3; they live stream makeup tutorials straight from red carpets, collaborating this past year with L'Oréal Paris at Cannes. Over the summer, they partnered with QVC to present, in real time, the looks Laura Geller was promoting during her live spot, so viewers could try them on as they watched, and — kaboom — make a more confident purchase decision.
AR is giving in-store shopping a boost, too, personalising the experience for customers. At Tom Ford’s newly opened beauty boutique in Covent Garden, customers can whiz through all 233 lipsticks in the Color Room (thanks, ModiFace), while, in the Makeup Room, they can record their makeup application sessions straight to their smartphone while gazing into the high-fidelity MemoMi “memory mirrors,” which also keep track of which products they tried.
In a true meeting of virtual and actual reality, Sephora’s Frivold notes that many of her customers find their favorite shades at home via Virtual Artist, and then come in to the store to try on their top three-to-five, “to get a sense for the formula, to see how they feel. It’s really an omni-channel shopping tool for us,” she says.
Customers also tinker with Virtual Tutorial, which gives step-by-step instruction, on their own faces, for everything from contouring to the evening smoky eye, complete with a shopping list of product recommendations. This past spring, Sephora incorporated Virtual Artist into the end-caps at two test stores, so customers could pick any lip or lash product off a shelf, scan the code on the box, and instantly see it on themselves. “The results blew us away — we’ve had over one million interactions since March,” says Frivold.
Even if customers’ experiences with augmented reality tools only sparked meaningful engagement with brands and their products — building awareness, buzz and excitement around new launches — that would be something of a win. But, does AR, where so many companies are investing so much, also translate to cold hard cash? It’s early days still, but the answer appears to be yes.
“We’ve seen significant evidence that it does drive sales,” says Aarabi. “The number we aim for is a 100-percent increase in sales, essentially to double conversions.” It’s a data point they successfully hit with participating Lauder brands (Clinique, Smashbox, Bobbi Brown and Estée Lauder) over Black Friday and Cyber Monday this past November. YouCam reports triple-digit growth during a test with an undisclosed brand at Selfridges this past year and double-digit growth when they bring their kiosks into department stores like Bloomingdale’s and Saks Fifth Avenue for special events.
The communication of selling is never going to be one-way again; people will interact using AR.
Data shows that when consumers use the try-on technology, they do buy, and often times, they buy more than they initially intended. Because it’s fun, and fast, AR also lures people out of their comfort zones, encouraging risk-free experimentation — like me, with my fuchsia liner and bronzy blush, two things I never would have considered otherwise. Jon Roman, vice president of Estée Lauder Online, reports that their users, who are trying an average of nine shades per session, are 25 percent more likely to add a product to their cart once they’ve used the Virtual Try-On tool. “They’re trying on multiple shades, and adding multiples to the cart,” he says. It’s quickly become a key part of their launch strategy when introducing new products, he says, citing last spring’s Pure Color Love lipstick as one example.
In addition to success with lip colour and eye shadow, Virtual Artist “has revolutionised shopping for false lashes,” Frivold says. “How else can you try on that product before you buy it?” Some users, she adds, try on lashes via the tool simply for the glamorous selfie the look provides. “What’s a better way to take a profile pic?”
An even bigger potential category than makeup, though, ventures Aarabi, is hair. “With makeup, if you get the wrong lipstick, it’s not the end of the world. If you get the wrong hair colour, it’s harder to go back.” This January, ModiFace will debut a live video 3D hair colouration tool where users — in salon with their colourist, or at home — can turn and twirl to see their hair from every angle. “It detects every single strand; it accounts for highlights, lowlights, ombré effects, and factors in your current hair colour,” says Aarabi. “It is still a slight deviation from reality,” he admits, “but it’s going to be quite close.” ModiFace’s launch partner is L’Oréal Professional, with other companies (“every major hair brand has signed on”) following.
And don’t forget the data collection angle. Roman says that the detailed analysis ModiFace supplies regarding what shades people try on based on skin tone, and what shades are most popular regionally, can not only help them personalise and promote what they’re offering, but also impacts how they stock inventory in individual doors. “They’re gathering that information for us right now so we can make it part of our regular way of doing business,” he says.
This is, of course, just the beginning. For one thing, “there is so much more that can be done with personalisation,” says Aarabi. Foundation matching, though it exists, remains a stumbling block, with everyone in the game racing to crack the code. In partnership with Google Home, Lauder is venturing into voice activation, to help consumers come up with tailored skin care regimens. YouCam is “moving full-force into AI,” says Gam, with a new Skin Diary that evaluates skin and tracks improvement, and a SKU-matching tool that will eventually be able to make product recommendations based on a shade swatch or magazine picture.
One thing is for sure, says Gam: “The communication of selling is never going to be one-way again; people will interact using AR.”
And when you think about it, the experience of peering into a magic mirror — “and seeing your reflection, with changes,” says Aarabi — is the eternal promise of the beauty industry, re-mastered for the digital age: You, only better.