Facebook's secretive Building 8 tested a prototype device to let you 'hear' through your skin

Robert Galbraith/flickr

• Starting as early as January 2017, the staff of a secretive Facebook initiative called Building 8 have been working to make the world's first brain-computer interfaces, devices that essentially put the functionality of a laptop in your head.
  
• The initiative includes at least two major publicly reported projects: a noninvasive brain sensor designed to turn thoughts into text and a device that essentially lets you "hear" with your skin.
• That second project is being led by Freddy Abnousi, a cardiologist who previously worked at Stanford University.
• In a study published this summer and reviewed by Business Insider, Abnousi and a team of 12 other researchers created a prototype device that includes a wearable armband that vibrates to turn the roots of words into silent speech.
• Facebook isn't alone in its brain-machine interface endeavor - Neuralink, a company founded by Elon Musk, is racing to achieve the same goal, and several startups have similar projects in the works.

When Regina Dugan, the former head of a secretive Facebook hardware lab known only as Building 8, took the stage at the company's annual developer conference in San Jose, California, last year, she announced the intent to build a device that few in the audience could believe would ever be real.

The device, she claimed, would let users hear "through their skin."

Hundreds of journalists were quick to dismiss the idea as science-fiction. But Dugan's initiative - now led by former Stanford University cardiologist Freddy Abnousi - appears to have turned into at least one prototype product. In a study published in July in a peer-reviewed engineering journal, a team of Facebook Building 8 researchers describe in detail, and with photographs, an armband that vibrates to allow for the roots of words to be transformed into silent speech.

Essentially, what the device appears to do is convert something that is heard - such as the sound of a news broadcast or a nearby conversation - into something that is felt in the form of vibration.

That could have a wide range of uses, from providing an alternative way (aside from American Sign Language) for members of the Deaf community to engage in conversation, to allowing someone to "listen" to something they are not permitted to hear, to allowing people to engage with a phone or computer while driving or doing some other activity.

"You could think of this as a very easy translation system where, instead of a Deaf person watching someone use American Sign Language to translate a speech, that person could simply wear one of these armbands," a neuroscientist at the University of California, San Francisco who spoke on condition of anonymity because he was not authorized to speak on behalf of the company, told Business Insider.

'A hardware device with many degrees of freedom'

A team of 12 researchers - half of them from Building 8 - wrote the paper outlining the device, which was published on July 31 in the peer-reviewed journal IEEE Transactions on Haptics (IEEE stands for Institute of Electrical and Electronics Engineers).

In addition to displaying a photo and diagram of the device, the authors describe in detail a series of tests they run on it using human study subjects. The subjects attempt to decipher what words the armband is communicating via various vibrating patterns.

IEEE

The device uses a concept called phonemes.

When we speak, the sounds we produce with our mouths can generally be broken into a smaller set of what are known as phonemes or root sounds. The word meat, for example, is composed of three phonemes: the sound that produces "M," the sound that produces "ee," and the sound that produces the hard "T."

The armband essentially turns each of those root sounds into a unique vibratory pattern. So the vibration for "M" would feel distinct from the vibration for "ee," and "T."

Every word then has its own unique vibratory pattern.

"This hardware device has many degrees of freedom," the UCSF neuroscientist said. "You have electrodes that vibrate and you can use that to create a symbol, and then take those symbols and map them onto speech."

The armband features a lot of electrodes, he said, as well as numerous ways to activate them. That gives users a lot of potential combinations to play around with - essentially, a fairly large potential vocabulary.

In the paper, 12 participants were trained to use the device by playing around with a keyboard until they got the gist of how it works. Then the armband was used to communicate 100 different words to them using 39 phonemes, and the participants were asked to identify the words.

Overall, the participants performed "pretty well," according to the UCSF neuroscientist. They not only learned how to use the device "fairly quickly," he said, but they were also "surprisingly good" at correctly identifying the words.

"All the participants were able to learn the haptic symbols associated with the 39 English phonemes with a 92% accuracy within 100 minutes," Abnousi and his coauthors write.

What the armband means for the race to link mind and machine

Several companies are currently racing to link mind and machine by way of devices called brain-computer interfaces. The first to put the functionality of a laptop in your head would pave the way for people to communicate seamlessly, instantly, and with whomever - or whatever - they want.

So far, two figures are publicly leading that race: Elon Musk and Mark Zuckerberg. Their clandestine projects, known as Neuralink and Building 8, respectively, focus on approaches that will require brain surgery, according to researchers familiar with their efforts.

But moonshots require small steps, and in order to create a brain-embedded computer, researchers must first re-think how we interact with our devices.

Building 8's two current semi-public projects include Abnousi's armband as well as a noninvasive brain sensor designed to turn thoughts into text. The brain-to-text sensor project, first described by former Business Insider journalist Alex Heath in April of last year, is being led by Mark Chevillet, a neuroscientist that Dugan hired in 2016. Dugan left Facebook in October of least year to "lead a new endeavor."

In an interview with Business Insider conducted last year, Abnousi described physical touch as "this innate way to communicate that we've been using for generations, but we've stepped away from it recently as we've become more screen-based."

Abnousi's aim? For his device to be "just part of you," he said.

Alex Heath contributed reporting.