Spelling out ​your thoughts

A man who was paralysed from the neck down in an accident more than a decade ago has written sentences using a computer system that turns imagined handwriting into words.

It is the first time that scientists have created sentences from brain activity linked to handwriting, and this paves the way for more sophisticated devices to help paralysed people communicate faster and more clearly.

The man, known as T5, who is in his 60s and lost practically all movement below his neck after a ​spinal cord injury in 2007, was able to write 18 words a minute when connected to the system. On individual letters, his “mindwriting” was more than 94 per cent accurate.

Imagined actions

Frank Willett, a research scientist on the project at Stanford University in California, said the approach opened the door to decoding other imagined actions, such as ten-finger touch typing and attempted speech for patients who had permanently lost their voices. “Instead of detecting letters, the algorithm would be detecting syllables, or rather phonemes, the fundamental unit of speech,” he said. Amy Orsborn, an expert in neural engineering at the University of Washington in Seattle, who was not involved in the work, called it “a remarkable advance” in the field.

Scientists have developed numerous software packages and devices to help paralysed people communicate, ranging from speech recognition programs to the muscle-driven cursor system created for the late Cambridge cosmologist Stephen Hawking (1942–2018), who used a screen on which a cursor automatically moved over the letters of the alphabet. To select one, and to build up words, he simply tensed his cheek.

Hawking’s system was a big improvement on the painstaking process used by Jean-Dominique Bauby (1952–1997), the late editor-in-chief of French Elle, to write his 1997 memoir, The Diving Bell and the Butterfly. Bauby had had a massive stroke in 1995 that left him “locked in”, able to blink only his left eyelid. He dictated the entire book by having the freelance editor Claude Mendibil read the alphabet aloud and blinking when she reached the next letter he wanted. For all the progress made since then, researchers have long dreamed of more efficient systems that tapped directly into the brain.

Brain-computer interfaces

In the case of T5, nine years after his accident, he enrolled on a clinical trial called BrainGate2 to investigate the safety of brain-computer interfaces (BCIs). These are small, implantable computer chips that read electrical activity straight from the brain. The man had two computer chips, each the size of a baby aspirin and bearing 100 electrodes, placed on the left side of his brain where neurons send signals to control the right hand.

Willett and his colleagues asked T5 to imagine he was holding a pen above a pad of paper and then to try to write individual letters of the alphabet, even though he could not move his arm or hand. As he tried, they recorded activity from the brain region that would have controlled his movements.

The scientists found that, more than ten years after the accident, the man’s brain still produced distinct patterns of neural activity for each letter and various punctuation marks. These recordings, and others taken as T5 tried to write out example sentences, were used to train an artificial intelligence algorithm. Many sessions later, the algorithm could predict in real time which letter the man was trying to write with 94.1 per cent accuracy. When the scientists added autocorrect, the accuracy rose to 99 per cent.

During the sessions, T5 often felt that the imaginary pen in his hand was moving across the page and tracing out letters, and that he could “write” faster if he kept letters small, Willett said. Some letters were harder to distinguish than others, according to the study in Nature. The letters “r”, “h” and “n”, for example, require similar movements and therefore similar brain activity.

Confused algorithms

One unknown is how well the algorithm might work for languages that do not use the Roman alphabet. For example, Tamil has 247 characters, many of which look similar, which could confuse the algorithm.

It will be some time before such brain-reading systems are ready for wider use. In an accompanying commentary, Orsborn and Pavithra Rajeswaran, also at the University of Washington, write that implantable BCIs “will need to provide tremendous performance and usability benefits to justify the expense and risks associated with implanting electrodes into the brain”.

Orsborn said: “This study is a remarkable advance for intracortical brain-computer interfaces because it achieves a huge leap in typing speed and accuracy.”

“While this gives us many reasons to be optimistic about the future of BCIs, challenges remain to develop systems that we can use for daily life. For instance, they demonstrated their algorithms in one person for a brief time. How we extend this work to assure the algorithms work well every day and for everyone is an exciting new challenge.”