WATCH: Paralysed woman speaks via digital avatar with help of AI in world's first
Artificial intelligence enables severely paralysed woman to speak after 18 years. Brain signals translated into speech and facial expressions.
A woman who was severely paralysed for 18 years has been able to speak with the help of artificial intelligence (AI) through a digital avatar. The latest technology translated her brain signals into speech and facial expressions, thus allowing her to communicate.
Before this advancement, patients relied on slow speech synthesisers that involved spelling out words using eye tracking or small facial movements, making natural conversation impossible, reported The Guardian.
About the patient
Ann, 47, has been severely paralysed since suffering a brainstem stroke more than 18 years ago. She cannot speak or type and normally communicates using movement-tracking technology that allows her to slowly select letters at up to 14 words a minute.
Ann hopes to work as a counsellor in the future with the help of avatar technology.
How does the technology work?
A paper-thin rectangle of tiny 253 electrodes is implanted on the surface of the patient's brain. This implant detects electrical activity in the part of the brain that controls speech and face movements. These signals are translated directly into a digital avatar’s speech and facial expressions including smiling, frowning, or surprise.
“Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others,” said Prof Edward Chang, who led the work at the University of California, San Francisco (UCSF). “These advancements bring us much closer to making this a real solution for patients.”
Training the AI algorithm and avatar
After the implantation of the device, Ann worked with the team to train the system’s AI algorithm to detect her unique brain signals for various speech sounds by repeating different phrases repeatedly.
The computer learned 39 sounds and a ChatGPT style model was created for making sentences from the signals. The avatar was then trained to sound like Ann based on her recording from a wedding before the injury.
How accurate were the results?
During a test run, words were decoded incorrectly 28% of the time when more than 500 phrases were checked. The system generated brain-to-text at a rate of 78 words a minute, compared with the 110-150 words typically spoken in natural conversation.
