From Stanford University Engineering : “New software turns ‘mental handwriting’ into words on computer screens”
From Stanford University Engineering
The new “mindwriting” technology enables a man with immobilized limbs to create text messages nearly as fast as people who use their thumbs to tap words onto smartphone keyboards.
New software and hardware taps into the brain to convert thoughts about handwriting into text on a computer screen. Credit: Aaron Burden/Unsplash
Stanford University investigators have coupled artificial intelligence software with a device, called a brain-computer interface (BCI), implanted in the brain of a man with full-body paralysis.
The software was able to decode information from the BCI to quickly convert the man’s thoughts about handwriting into text on a computer screen.
The man was able to write using this approach more than twice as quickly as he could using a previous method developed by the Stanford researchers, who reported those findings in 2017 in the journal eLife.
The new findings, published online May 12 in Nature,could spur further advances benefiting hundreds of thousands of Americans, and millions globally, who’ve lost the use of their upper limbs or their ability to speak due to spinal-cord injuries, strokes or amyotrophic lateral sclerosis, also known as Lou Gehrig’s disease, said Jaimie Henderson, professor of neurosurgery.
“This approach allowed a person with paralysis to compose sentences at speeds nearly comparable to those of able-bodied adults of the same age typing on a smartphone,” said Henderson, the John and Jene Blume–Robert and Ruth Halperin Professor at Stanford Medicine. “The goal is to restore the ability to communicate by text.”
The participant in the study produced text at a rate of about 18 words per minute. By comparison, able-bodied people of the same age can punch out about 23 words per minute on a smartphone.
The participant, referred to as T5, lost practically all movement below the neck because of a spinal-cord injury in 2007. Nine years later, Henderson placed two brain-computer-interface chips, each the size of a baby aspirin, on the left side of T5’s brain. Each chip has 100 electrodes that pick up signals from neurons firing in the part of the motor cortex – a region of the brain’s outermost surface – that governs hand movement.
Those neural signals are sent via wires to a computer, where artificial intelligence algorithms decode the signals and surmise T5’s intended hand and finger motion. The algorithms were designed in Stanford’s Neural Prosthetics Translational Laboratory, co-directed by Henderson and Krishna Shenoy, the Hong Seh and Vivian W. M. Lim Professor in the School of Engineering and professor of electrical engineering.
Shenoy and Henderson, who have been collaborating on BCIs since 2005, are the senior co-authors of the new study. The lead author is Frank Willett, a research scientist in the Neural Prosthetics Translational Laboratory and with the Howard Hughes Medical Institute (HHMI) (US).
“We’ve learned that the brain retains its ability to prescribe fine movements a full decade after the body has lost its ability to execute those movements,” Willett said. “And we’ve learned that complicated intended motions involving changing speeds and curved trajectories, like handwriting, can be interpreted more easily and more rapidly by the artificial intelligence algorithms we’re using than can simpler intended motions like moving a cursor in a straight path at a steady speed. Alphabetical letters are different from one another, so they’re easier to tell apart.”
In the 2017 study, three participants with limb paralysis, including T5 – all with BCIs placed in the motor cortex – were asked to concentrate on using an arm and hand to move a cursor from one key to the next on a computer-screen keyboard display, then to focus on clicking on that key.
In that study, T5 set what was until now the all-time record: copying displayed sentences at about 40 characters per minute. Another study participant was able to write extemporaneously, selecting whatever words she wanted, at 24.4 characters per minute.
If the paradigm underlying the 2017 study was analogous to typing, the model for the new Nature study is analogous to handwriting. T5 concentrated on trying to write individual letters of the alphabet on an imaginary legal pad with an imaginary pen, despite his inability to move his arm or hand. He repeated each letter 10 times, permitting the software to “learn” to recognize the neural signals associated with his effort to write that particular letter.
In numerous multi-hour sessions that followed, T5 was presented with groups of sentences and instructed to make a mental effort to “handwrite” each one. No uppercase letters were employed. Examples of the sentences were “i interrupted, unable to keep silent,” and “within thirty seconds the army had landed.” Over time, the algorithms improved their ability to differentiate among the neural firing patterns typifying different characters. The algorithms’ interpretation of whatever letter T5 was attempting to write appeared on the computer screen after a roughly half-second delay.
In further sessions, T5 was instructed to copy sentences the algorithms had never been exposed to. He was eventually able to generate 90 characters, or about 18 words, per minute. Later, asked to give his answers to open-ended questions, which required some pauses for thought, he generated 73.8 characters (close to 15 words, on average) per minute, tripling the previous free-composition record set in the 2017 study.
T5’s sentence-copying error rate was about one mistake in every 18 or 19 attempted characters. His free-composition error rate was about one in every 11 or 12 characters. When the researchers used an after-the-fact autocorrect function – similar to the ones incorporated into our smartphone keyboards – to clean things up, those error rates were markedly lower: below 1% for copying, and just over 2% for freestyle.
These error rates are quite low compared with other BCIs, said Shenoy, who is also a Howard Hughes Medical Institute investigator.
“While handwriting can approach 20 words per minute, we tend to speak around 125 words per minute, and this is another exciting direction that complements handwriting. If combined, these systems could together offer even more options for patients to communicate effectively,” Shenoy said.
See the full article here .
five-ways-keep-your-child-safe-school-shootings
Please help promote STEM in your local schools.
Stanford Engineering has been at the forefront of innovation for nearly a century, creating pivotal technologies that have transformed the worlds of information technology, communications, health care, energy, business and beyond.
The school’s faculty, students and alumni have established thousands of companies and laid the technological and business foundations for Silicon Valley. Today, the school educates leaders who will make an impact on global problems and seeks to define what the future of engineering will look like.
Mission
Our mission is to seek solutions to important global problems and educate leaders who will make the world a better place by using the power of engineering principles, techniques and systems. We believe it is essential to educate engineers who possess not only deep technical excellence, but the creativity, cultural awareness and entrepreneurial skills that come from exposure to the liberal arts, business, medicine and other disciplines that are an integral part of the Stanford experience.
Our key goals are to:
Conduct curiosity-driven and problem-driven research that generates new knowledge and produces discoveries that provide the foundations for future engineered systems
Deliver world-class, research-based education to students and broad-based training to leaders in academia, industry and society
Drive technology transfer to Silicon Valley and beyond with deeply and broadly educated people and transformative ideas that will improve our society and our world.
The Future of Engineering
The engineering school of the future will look very different from what it looks like today. So, in 2015, we brought together a wide range of stakeholders, including mid-career faculty, students and staff, to address two fundamental questions: In what areas can the School of Engineering make significant world‐changing impact, and how should the school be configured to address the major opportunities and challenges of the future?
One key output of the process is a set of 10 broad, aspirational questions on areas where the School of Engineering would like to have an impact in 20 years. The committee also returned with a series of recommendations that outlined actions across three key areas — research, education and culture — where the school can deploy resources and create the conditions for Stanford Engineering to have significant impact on those challenges.
Leland and Jane Stanford founded the University to “promote the public welfare by exercising an influence on behalf of humanity and civilization.” Stanford opened its doors in 1891, and more than a century later, it remains dedicated to finding solutions to the great challenges of the day and to preparing our students for leadership in today’s complex world. Stanford, is an American private research university located in Stanford, California on an 8,180-acre (3,310 ha) campus near Palo Alto. Since 1952, more than 54 Stanford faculty, staff, and alumni have won the Nobel Prize, including 19 current faculty members
sciencesprings 
Reply