Students Add Facial Recognition to Meta Smart Glasses to Identify Strangers in Real-Time

Two students at Harvard University modified a set of Meta Ray-Ban Smart Glasses to add AI-based facial recognition technology, and the result is a wearable that is able to identify people in real time, according to a report from 404 Media.

The modified ‌Meta‌ Glasses are able to use the camera to scan a stranger's face to get their name, and there's also a function for pulling information like home address, phone number, and family members from internet sites that store that kind of data.

AnhPhu Nguyen and Caine Ardayfio, the creators, demonstrated the glasses in public. Ardayfio walked up to a total stranger, scanned her face, got her name and affiliations, and then pretended that he knew her from one of the organizations that came up in a search about her. In a second demo, Nguyen identified a random man and initiated a conversation, pretending to have read his work based on what the glasses surfaced.

The ‌Meta‌ Glasses were set up to stream video directly to Instagram, and a computer program monitors the stream. AI determines when a face is detected, and then an internet search is conducted to find more pictures of the person. From there, online articles, voter registration databases, and other public information is used to suss out a name, phone number, address, and relative's names. Everything happens in a couple of minutes, and it's fed back to a smartphone app.

Dozens of Harvard students were identified and they were never aware that AI face scans had been used to suss out their names and information about them. Nguyen said that the project is meant to raise awareness of what can be done with today's consumer technology, face search engines, LLMs, and public databases. The two creators have no plans to release any kind of product or code.

Nguyen and Ardayfio chose the ‌Meta‌ Ray-Ban Glasses because "they look almost indistinguishable from regular glasses," making them ideal for surreptitious image capture. Reverse facial recognition was used to match faces to publicly available images, and PimEyes, a face recognition search engine, was identified as the "most effective" way to find people. An LLM was used to look up details about a person across the web, and FastPeopleSearch provided additional information. The entire setup is automatic, thanks to LLMs.

‌Meta‌'s Ray-Ban Glasses are simple in design and are primarily focused on taking images, but even that limited hardware is enough to generate detailed information about a person almost instantly. With a faster processor, on-device AI capabilities, better camera technology, and augmented reality, future products could do even more.

In response to the report, ‌Meta‌ told 404 Media that PimEyes facial recognition can be used with any camera, and it's not something that is "only possible because of ‌Meta‌ Ray-Bans." That is true, but the ‌Meta‌ Ray-Bans look like a standard pair of glasses, and many people aren't aware when the glasses are being used to take photos. PimEyes said that it found the report "quite surprising" because its search engine is "not designed to identify individuals, either directly or indirectly."