MIT’s Cheetah 3 robot — an upgrade to the Cheetah 2, can now leap and gallop across rough terrain, climb a staircase littered with debris, and quickly recover its balance when suddenly yanked or shoved — all while essentially blind.
The 90-pound robot is intentionally designed to do all this without relying on cameras or any external environmental sensors. The idea is to allow it to “feel” its way through its surroundings via “blind locomotion,” (like making your way across a pitch-black room), eliminating visual distractions, which would slow the robot down.
“Vision can be noisy, slightly inaccurate, and sometimes not available, and if you rely too much on vision, your robot has to be very accurate in position and eventually will be slow, said the robot’s designer, Sangbae Kim, associate professor of mechanical engineering at MIT. “So we want the robot to rely more on tactile information. That way, it can handle unexpected obstacles while moving fast.”
Faster, more nimble, more cat-like
Warning: Cheetah 3 can jump on your desk (credit: MIT)
Cheetah3 has an expanded range of motion compared to its predecessor Cheetah 2, which allows the robot to stretch backwards and forwards, and twist from side to side, much like a cat limbering up to pounce. Cheetah 3 can blindly make its way up staircases and through unstructured terrain, and can quickly recover its balance in the face of unexpected forces, thanks to two new algorithms developed by Kim’s team: a contact detection algorithm, and a model-predictive control algorithm.
The contact detection algorithm helps the robot determine the best time for a given leg to switch from swinging in the air to stepping on the ground. For example, if the robot steps on a light twig versus a hard, heavy rock, how it reacts — and whether it continues to carry through with a step, or pulls back and swings its leg instead — can make or break its balance.
The researchers tested the algorithm in experiments with the Cheetah 3 trotting on a laboratory treadmill and climbing on a staircase. Both surfaces were littered with random objects such as wooden blocks and rolls of tape.
The robot’s blind locomotion was also partly due to the model-predictive control algorithm, which predicts how much force a given leg should apply once it has committed to a step. The model-predictive control algorithm calculates the multiplicative positions of the robot’s body and legs a half-second into the future, if a certain force is applied by any given leg as it makes contact with the ground.
Cameras to be activated later
The team had already added cameras to the robot to give it visual feedback of its surroundings. This will help in mapping the general environment, and will give the robot a visual heads-up on larger obstacles such as doors and walls. But for now, the team is working to further improve the robot’s blind locomotion
“We want a very good controller without vision first,” Kim says. “And when we do add vision, even if it might give you the wrong information, the leg should be able to handle [obstacles]. Because what if it steps on something that a camera can’t see? What will it do? That’s where blind locomotion can help. We don’t want to trust our vision too much.”
Within the next few years, Kim envisions the robot carrying out tasks that would otherwise be too dangerous or inaccessible for humans to take on.
This research was supported, in part, by Naver, Toyota Research Institute, Foxconn, and Air Force Office of Scientific Research.
A “mini arm” made up of two hinges of actuating nickel hydroxide-oxyhydroxide material (left) can lift an object 50 times its own weight when triggered (right) by light or electricity. (credit: University of Hong Kong)
University of Hong Kong researchers have invented a radical new lightweight material that could replace traditional bulky, heavy motors or actuators in robots, medical devices, prosthetic muscles, exoskeletons, microrobots, and other types of devices.
The new actuating material — nickel hydroxide-oxyhydroxide — can be instantly triggered and wirelessly powered by low-intensity visible light or electricity at relatively low intensity. It can exert a force of up to 3000 times its own weight — producing stress and speed comparable to mammalian skeletal muscles, according to the researchers.
The material is also responsive to heat and humidity changes, which could allow autonomous machines to harness tiny energy changes in the environment.
The major component is nickel, so the material cost is low, and fabrication uses a simple electrodeposition process, allowing for scaling up and manufacture in industry.
Developing actuating materials was identified as the leading grand challenge in “The grand challenges of Science Robotics” to “deeply root robotics research in science while developing novel robotic platforms that will enable new scientific discoveries.”
Using a light blocker (top) a mini walking bot (bottom) with the “front leg” bent and straightened alternatively can walk towards a light source. (credit: University of Hong Kong)
University of Hong Kong | Future Robots need No Motors
MIT Media Lab (no sound) | Intro: Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy. This is an example of a therapy session augmented with SoftBank Robotics’ humanoid robot NAO and deep-learning software. The 35 children with autism who participated in this study ranged in age from 3 to 13. They reacted in various ways to the robots during their 35-minute sessions — from looking bored and sleepy in some cases to jumping around the room with excitement, clapping their hands, and laughing or touching the robot.
Robots armed with personalized “deep learning” software could help therapists interpret behavior and personalize therapy of autistic children, while making the therapy more engaging and natural. That’s the conclusion of a study by an international team of researchers at MIT Media Lab, Chubu University, Imperial College London, and University of Augsburg.*
Children with autism-spectrum conditions often have trouble recognizing the emotional states of people around them — distinguishing a happy face from a fearful face, for instance. So some therapists use a kid-friendly robot to demonstrate those emotions and to engage the children in imitating the emotions and responding to them in appropriate ways.
Personalized autism therapy
But the MIT research team realized that deep learning would help the therapy robots perceive the children’s behavior more naturally, they report in a Science Robotics paper.
Personalization is especially important in autism therapy, according to the paper’s senior author, Rosalind Picard, PhD, a professor at MIT who leads research in affective computing: “If you have met one person, with autism, you have met one person with autism,” she said, citing a famous adage.
“Computers will have emotional intelligence by 2029”… by which time, machines will “be funny, get the joke, and understand human emotion.” — Ray Kurzweil
“The challenge of using AI [artificial intelligence] that works in autism is particularly vexing, because the usual AI methods require a lot of data that are similar for each category that is learned,” says Picard, in explaining the need for deep learning. “In autism, where heterogeneity reigns, the normal AI approaches fail.”
How personalized robot-assisted therapy for autism would work
Robot-assisted therapy** for autism often works something like this: A human therapist shows a child photos or flash cards of different faces meant to represent different emotions, to teach them how to recognize expressions of fear, sadness, or joy. The therapist then programs the robot to show these same emotions to the child, and observes the child as she or he engages with the robot. The child’s behavior provides valuable feedback that the robot and therapist need to go forward with the lesson.
“Therapists say that engaging the child for even a few seconds can be a big challenge for them. [But] robots attract the attention of the child,” says lead author Ognjen Rudovic, PhD, a postdoctorate fellow at the MIT Media Lab. “Also, humans change their expressions in many different ways, but the robots always do it in the same way, and this is less frustrating for the child because the child learns in a very structured way how the expressions will be shown.”
SoftBank Robotics | The researchers used NAO humanoid robots in this study. Almost two feet tall and resembling an armored superhero or a droid, NAO conveys different emotions by changing the color of its eyes, the motion of its limbs, and the tone of its voice.
However, this type of therapy would work best if the robot could also smoothly interpret the child’s own behavior — such as excited or paying attention — during the therapy, according to the researchers. To test this assertion, researchers at the MIT Media Lab and Chubu Universitydeveloped a personalized deep learning network that helps robots estimate the engagement and interest of each child during these interactions, they report.**
The researchers built a personalized framework that could learn from data collected on each individual child. They captured video of each child’s facial expressions, head and body movements, poses and gestures, audio recordings and data on heart rate, body temperature, and skin sweat response from a monitor on the child’s wrist.
Most of the children in the study reacted to the robot “not just as a toy but related to NAO respectfully, as it if was a real person,” said Rudovic, especially during storytelling, where the therapists asked how NAO would feel if the children took the robot for an ice cream treat.
In the study, the researchers found that the robots’ perception of the children’s responses agreed with assessments by human experts with a high correlation score of 60 percent, the scientists report.*** (It can be challenging for human observers to reach high levels of agreement about a child’s engagement and behavior. Their correlation scores are usually between 50 and 55 percent, according to the researchers.)
* The study was funded by grants from the Japanese Ministry of Education, Culture, Sports, Science and Technology; Chubu University; and the European Union’s HORIZON 2020 grant (EngageME).
** A deep-learning system uses hierarchical, multiple layers of data processing to improve its tasks, with each successive layer amounting to a slightly more abstract representation of the original raw data. Deep learning has been used in automatic speech and object-recognition programs, making it well-suited for a problem such as making sense of the multiple features of the face, body, and voice that go into understanding a more abstract concept such as a child’s engagement.
Overview of the key stages (sensing, perception, and interaction) during robot-assisted autism therapy. Data from three modalities (audio, visual, and autonomic physiology) were recorded using unobtrusive audiovisual sensors and sensors worn on the child’s wrist, providing the child’s heart-rate, skin-conductance (EDA), body temperature, and accelerometer data. The focus of this work is the robot perception, for which we designed the personalized deep learning framework that can automatically estimate levels of the child’s affective states and engagement. These can then be used to optimize the child-robot interaction and monitor the therapy progress (see Interpretability and utility). The images were obtained by using Softbank Robotics software for the NAO robot. (credit: Ognjen Rudovic et al./Science Robotics)
“In the case of facial expressions, for instance, what parts of the face are the most important for estimation of engagement?” Rudovic says. “Deep learning allows the robot to directly extract the most important information from that data without the need for humans to manually craft those features.”
The robots’ personalized deep learning networks were built from layers of these video, audio, and physiological data, information about the child’s autism diagnosis and abilities, their culture and their gender. The researchers then compared their estimates of the children’s behavior with estimates from five human experts, who coded the children’s video and audio recordings on a continuous scale to determine how pleased or upset, how interested, and how engaged the child seemed during the session.
*** Trained on these personalized data coded by the humans, and tested on data not used in training or tuning the models, the networks significantly improved the robot’s automatic estimation of the child’s behavior for most of the children in the study, beyond what would be estimated if the network combined all the children’s data in a “one-size-fits-all” approach, the researchers found. Rudovic and colleagues were also able to probe how the deep learning network made its estimations, which uncovered some interesting cultural differences between the children. “For instance, children from Japan showed more body movements during episodes of high engagement, while in Serbs large body movements were associated with disengagement episodes,” Rudovic notes.
This tampered image (left) can be detected by noting visual artifacts (red rectangle, showing the unnaturally high contrast along the baseball player’s edges), compared to authentic regions (the parking lot background); and by noting noise pattern inconsistencies between the tampered regions and the background (as seen in “Noise” image). The “ground-truth” image is the outline of the added (fake) image used in the experiment. (credit: Adobe)
Thanks to user-friendly image editing software like Adobe Photoshop, it’s becoming increasingly difficult and time-consuming to spot some deceptive image manipulations.
Now, funded by DARPA, researchers at Adobe and the University of Maryland, College Park have turned to AI to detect the more subtle methods now used in doctoring images.
Forensic AI
What used to take an image-forensic expert several hours to do can now be done in seconds with AI, says Vlad Morariu, PhD, a senior research scientist at Adobe. “Using tens of thousands of examples of known, manipulated images, we successfully trained a deep learning neural network* to recognize image manipulation in each image,” he explains.
“We focused on three common tampering techniques — splicing, where parts of two different images are combined; copy-move, where objects in a photograph are moved or cloned from one place to another; and removal, where an object is removed from a photograph, and filled-in,” he notes.
The neural network looks for two things: changes to the red, green and blue color values of pixels; and inconsistencies in the random variations of color and brightness generated by a camera’s sensor or by later software manipulations, such as Gaussian smoothing.
* The researchers used a “two-stream Faster R-CNN” (a type of convolutional neural network) that they trained end-to-end to detect the tampered regions in a manipulated image. The two streams are RGB (red-green-blue — the millions of different colors) to find tampering artifacts like strong contrast difference and unnatural tampered boundaries; and noise (inconsistency of noise patterns between authentic and tampered regions — in the example above, note that the baseball player’s image is lighter, for example, in addition to more-subtle differences that can be detected by the algorithm — even what tampering technique was used). These two features are then fused together to further identify spatial co-occurrence of these two modalities (RGB and noise).
A user supervises and controls an autonomous robot using brain signals to detect mistakes and muscle signals to redirect a robot in a task to move a power drill to one of three possible targets on the body of a mock airplane. (credit: MIT)
Getting robots to do things isn’t easy. Usually, scientists have to either explicitly program them, or else train them to understand human language. Both options are a lot of work.
Now a new system developed by researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Vienna University of Technology, and Boston University takes a simpler approach: It uses a human’s brainwaves and hand gestures to instantly correct robot mistakes.
Plug and play
Instead of trying to mentally guide the robot (which would require a complex, error-prone system and extensive operator training), the system identifies robot errors in real time by detecting a specific type of electroencephalogram (EEG) signal called “error-related potentials,” using a brain-computer interface (BCI) cap. These potentials (voltage spikes) are unconsciously produced in the brain when people notice mistakes — no user training required.
If an error-related potential signal is detected, the system automatically stops. That allows the supervisor to correct the robot by simply flicking a wrist — generating an electromyogram (EMG) signal that is detected by a muscle sensor in the supervisor’s arm to provide specific instructions to the robot.*
To develop the system, the researchers used “Baxter,” a popular humanoid robot from Rethink Robotics, shown here folding a shirt. (credit: Rethink Robotics)
Remarkably, the “plug and play” system works without requiring supervisors to be trained. So organizations could easily deploy it in real-world use in manufacturing and other areas. Supervisors can even manage teams of robots.**
For the project, the team used “Baxter,” a humanoid robot from Rethink Robotics. With human supervision, the robot went from choosing the correct target 70 percent of the time to more than 97 percent of the time in a multi-target selection task for a mock drilling operation.
“This work combining EEG and EMG feedback enables natural human-robot interactions for a broader set of applications than we’ve been able to do before using only EEG feedback,” says CSAIL Director Daniela Rus, who supervised the work. “By including muscle feedback, we can use gestures to command the robot spatially, with much more nuance and specificity.”
“A more natural and intuitive extension of us”
The team says that they could imagine the system one day being useful for the elderly, or workers with language disorders or limited mobility.
“We’d like to move away from a world where people have to adapt to the constraints of machines,” says Rus. “Approaches like this show that it’s very much possible to develop robotic systems that are a more natural and intuitive extension of us.”
* EEG and EMG both have some individual shortcomings: EEG signals are not always reliably detectable, while EMG signals can sometimes be difficult to map to motions that are any more specific than “move left or right.” Merging the two, however, allows for more robust bio-sensing and makes it possible for the system to work on new users without training.
** The “plug and play” supervisory control system
“If an [error-related potential] or a gesture is detected, the robot halts and requests assistance. The human then gestures to the left or right to naturally scroll through possible targets. Once the correct target is selected, the robot resumes autonomous operation. … The system includes an experiment controller and the Baxter robot as well as EMG and EEG data acquisition and classification systems. A mechanical contact switch on the robot’s arm detects initiation of robot arm motion. A human supervisor closes the loop.” — Joseph DelPreto et al. Plug-and-Play Supervisory Control Using Muscle and Brain Signals for Real-Time Gesture and Error Detection. Robotics: Science and Systems Proceedings (forthcoming). (credit: MIT)
Crossbar arrays of non-volatile memories can accelerate the training of neural networks by performing computation at the actual location of the data. (credit: IBM Research)
Imagine advanced artificial intelligence (AI) running on your smartphone — instantly presenting the information that’s relevant to you in real time. Or a supercomputer that requires hundreds of times less energy.
The IBM Research AI team has demonstrated a new approach that they believe is a major step toward those scenarios.
Deep neural networks normally require fast, powerful graphical processing unit (GPU) hardware accelerators to support the needed high speed and computational accuracy — such as the GPU devices used in the just-announced Summit supercomputer. But GPUs are highly energy-intensive, making their use expensive and limiting their future growth, the researchers explain in a recent paper published in Nature.
Analog memory replaces software, overcoming the “von Neumann bottleneck”
Instead, the IBM researchers used large arrays of non-volatile analog memory devices (which use continuously variable signals rather than binary 0s and 1s) to perform computations. Those arrays allowed the researchers to create, in hardware, the same scale and precision of AI calculations that are achieved by more energy-intensive systems in software, but running hundreds of times faster and at hundreds of times lower power — without sacrificing the ability to create deep learning systems.*
The trick was to replace conventional von Neumann architecture, which is “constrained by the time and energy spent moving data back and forth between the memory and the processor (the ‘von Neumann bottleneck’),” the researchers explain in the paper. “By contrast, in a non-von Neumann scheme, computing is done at the location of the data [in memory], with the strengths of the synaptic connections (the ‘weights’) stored and adjusted directly in memory.
“Delivering the future of AI will require vastly expanding the scale of AI calculations,” they note. “Instead of shipping digital data on long journeys between digital memory chips and processing chips, we can perform all of the computation inside the analog memory chip. We believe this is a major step on the path to the kind of hardware accelerators necessary for the next AI breakthroughs.”**
Given these encouraging results, the IBM researchers have already started exploring the design of prototype hardware accelerator chips, as part of an IBM Research Frontiers Institute project, they said.
* “From these early design efforts, we were able to provide, as part of our Nature paper, initial estimates for the potential of such [non-volatile memory]-based chips for training fully-connected layers, in terms of the computational energy efficiency (28,065 GOP/sec//W) and throughput-per-area (3.6 TOP/sec/mm2). These values exceed the specifications of today’s GPUs by two orders of magnitude. Furthermore, fully-connected layers are a type of neural network layer for which actual GPU performance frequently falls well below the rated specifications. … Analog non-volatile memories can efficiently accelerate at the heart of many recent AI advances. These memories allow the “multiply-accumulate” operations used throughout these algorithms to be parallelized in the analog domain, at the location of weight data, using underlying physics. Instead of large circuits to multiply and add digital numbers together, we simply pass a small current through a resistor into a wire, and then connect many such wires together to let the currents build up. This lets us perform many calculations at the same time, rather than one after the other.”
** “By combining long-term storage in phase-change memory (PCM) devices, near-linear update of conventional complementary metal-oxide semiconductor (CMOS) capacitors and novel techniques for cancelling out device-to-device variability, we finessed these imperfections and achieved software-equivalent DNN accuracies on a variety of different networks. These experiments used a mixed hardware-software approach, combining software simulations of system elements that are easy to model accurately (such as CMOS devices) together with full hardware implementation of the PCM devices. It was essential to use real analog memory devices for every weight in our neural networks, because modeling approaches for such novel devices frequently fail to capture the full range of device-to-device variability they can exhibit.”
Huge self-driving-car video dataset may help reduce accidents
Berkeley Deep Drive, the largest-ever self-driving car dataset, has been released by BDD Industry Consortium for free public download. It features 100,000 HD videos on cars and labeled objects, with GPS and other data — 800 times larger than Baidu’s Apollo dataset. The goal: apply computer vision research — including deep reinforcement learning for object tracking — to the automotive field.
Berkeley researchers plan to add to the dataset, including panorama and stereo videos, LiDAR, and radar. Ref.: arXiv. Source: BDD Industry Consortium.
A “privacy filter” that disrupts facial-recognition algorithms. A “difference” filter alters very specific pixels in the image, making subtle changes (such as in the corner of the eyes). (credit:Avishek Bose)
A “privacy filter” for photos
University of Toronto engineering researchers have created an artificial intelligence (AI) algorithm (computer program) to disrupt facial recognition systems and protect privacy. It uses a deep-learning technique called “adversarial training,” which pits two algorithms against each other — one to identify faces, and the second to disrupt the facial recognition task of the first.
The algorithm also disrupts image-based search, feature identification, emotion, and ethnicity estimation, and all other face-based attributes that can be extracted automatically. It will be available as an app or website. Ref.: Github. Source: University of Toronto.
Developers of the more than 2 million iOS apps will be able to hook into Siri’s new Suggestions feature, with help from a new “Create ML” tool. (credit: TechCrunch)
A smarter Siri
“Apple is turning its iPhone into a highly personalized device, powered by its [improved] Siri AI,” says TechCrunch, reporting on the just-concluded Apple Worldwide Developers Conference. With the new “Suggestions” feature — to be available with Apple’s iOS 12 mobile operating system (in autumn 2018) — Siri will offer suggestions to users, such as texting someone that you’re running late to a meeting.
The Photos app will also get smarter, with a new tab that will “prompt users to share photos taken with other people, thanks to facial recognition and machine learning,” for example, says TechCrunch. Along with Core ML (announced last year), a new tool called “Create ML” should help Apple developers build machine learning models, reports Wired.
(credit: Loughborough University)
AI detects illnesses in human breath
Researchers at Loughborough University in the U.K. have developed deep-learning networks that can detect illness-revealing chemical compounds in breath samples, with potentially wide applications in medicine, forensics, environmental analysis, and others.
The new process is cheaper and more reliable — taking only minutes to autonomously analyze a breath sample that previously took hours by a human expert, using gas-chromatography mass-spectrometers (GC-MS). The initial study focused on recognizing a group of chemicals called aldehydes, which are often associated with fragrances but also human stress conditions and illnesses. Source: The Conversation.
MIT’s Sims-inspired “VirtualHome” system aims to teach artificial agents a range of chores, such as setting the table and making coffee. (credit: MIT CSAIL)
Computer scientists at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and the University of Toronto* have created a Sims-inspired “VirtualHome” system that can simulate detailed household tasks.
The idea is to allow “artificial agents” to execute tasks — opening up the possibility of one day teaching robots to do such tasks.
Using crowdsourcing, the researchers created videos that simulate detailed household activities and sub-tasks in eight different scenes, including a living room, kitchen, dining room, bedroom, and home office. A simple model can generate a program from either a video or a textual description, allowing robots to be programmed by naive users, either via natural language or video demonstration.
“Hey, Jeeves — get me a glass of milk” would require several subtasks — the first five are shown here. (credit: MIT CSAIL)
The researchers have trained the system using nearly 3,000 programs for various activities, which are further broken down into subtasks for the computer to understand. A simple task like “making coffee,” for example, would also include the necessary step, “grabbing a cup.”
Next: Anticipating personalized wants and needs
The end result is a large database of household tasks described using natural language. Companies like Amazon that are working to develop Alexa-like robotic systems at home could in the future use such data to train their models to do more complex tasks.
Robots could eventually be trained to anticipate personalized wants and needs, which could be especially helpful as assistive technology for the elderly, or for those with limited mobility.
The team hopes to train the robots using actual videos instead of Sims-style simulation videos. That would enable a robot to learn directly by simply watching a YouTube video. The team is also working on a reward-learning system in which the robot gets positive feedback when it does tasks correctly.
* Researchers from McGill University and the University of Ljubljana were also involved.
** This project was partially supported by a “La Caixa” fellowship, Canada’s National Sciences and Engineering Research Council Strategic Partnership Network on Machine Learning Hardware Acceleration (NSERC COHESA), Samsung, the Defense Advanced Research Projects Agency (DARPA) and the Intelligence Advanced Research Projects Activity (IARPA).
A self-healing material that spontaneously repairs itself in real time from extreme mechanical damage, such as holes cut in it multiple times. New pathways are formed instantly and autonomously to keep this circuit functioning and the device moving. (credit: Carnegie Mellon University College of Engineering)
Carnegie Mellon University (CMU) researchers have created a self-healing material that spontaneously repairs itself under extreme mechanical damage, similar to many natural organisms. Applications include bio-inspired first-responder robots that instantly heal themselves when damaged and wearable computing devices that recover from being dropped.
The new material is composed of liquid metal droplets suspended in a soft elastomer (a material with elastic properties, such as rubber). When damaged, the droplets rupture to form new connections with neighboring droplets, instantly rerouting electrical signals. Circuits produced with conductive traces of this material remain fully and continuously operational when severed, punctured, or have material removed.
“Other research in soft electronics has resulted in materials that are elastic, but are still vulnerable to mechanical damage that causes immediate electrical failure,” said Carmel Majidi, PhD, a CMU associate professor of mechanical engineering, who also directs the Integrated Soft Materials Laboratory. “The unprecedented level of functionality of our self-healing material can enable soft-matter electronics and machines to exhibit the extraordinary resilience of soft biological tissue and organisms.”
The self-healing material also exhibits high ability to conduct electricity, which is not affected when stretched. That makes it ideal for uses in power and data transmission, as a health-monitoring device on an athlete during rigorous training, or an inflatable structure that can withstand environmental extremes on Mars, for example.
Noninvasive brain–computer interface (BCI) systems can restore functions lost to disability — allowing for spontaneous, direct brain control of external devices without the risks associated with surgical implantation of neural interfaces. But as machine-learning algorithms have become faster and more powerful, researchers have mostly focused on increasing performance by optimizing pattern-recognition algorithms.
But what about letting patients actively participate with AI in improving performance?
To test that idea, researchers at the École Polytechnique Fédérale de Lausanne (EPFL), based in Geneva, Switzerland, conducted research using “mutual learning” between computer and humans — two severely impaired (tetraplegic) participants with chronic spinal cord injury. The goal: win a live virtual racing game at an international event.
Controlling a racing-game avatar using a BCI
A computer graphical user interface for the race track in Cybathlon 2016 “Brain Runners“ game. “Pilots” (participants) had to deliver (by thinking) the proper command in each color pad (cyan, magenta, yellow) to accelerate their own avatar in the race. (credit: Serafeim Perdikis and Robert Leeb)
The participants were trained to improve control of an avatar (a person-substitute shown on a computer screen) in a virtual racing game. The experiment used a brain-computer interface (BCI), which uses electrodes on the head to pick up control signals from a person’s brain.
Each participant (called a “pilot”) controlled an on-screen avatar in a three-part race. This required mastery of separate commands for spinning, jumping, sliding, and walking without stumbling.
After training for several months, in Oct. 8, 2016, the two pilots participated (on the “Brain Tweakers” team) in Cybathlon in Zurich, Switzerland — the first international para-Olympics for disabled individuals in control of bionic assistive technology.*
The BCI-based race consisted of four brain-controlled avatars competing in a virtual racing game called “Brain Runners.” To accelerate each pilot’s avatar, they had to issue up to three mental commands (or intentional idling) on corresponding color-coded track segments.
Maximizing BCI performance by humanizing mutual learning
The researchers believe that with the mutual-learning approach, they have “maximized the chances for human learning by infrequent recalibration of the computer, leaving time for the human to better learn how to control the sensorimotor rhythms that would most efficiently evoke the desired avatar movement. Our results showcase strong and continuous learning effects at all targeted levels — machine, subject, and application — with both [participants] over a longitudinal study lasting several months,” the researchers conclude.
Reference (open-source): PLoS Biology May 10, 2018
* At Cybathlon, each team comprised a pilot together with scientists and technology providers of the functional and assistive devices used, which can be prototypes developed by research labs or companies, or commercially available products. That also makes Cybathlon a competition between companies and research laboratories. The next Cybathlon will be held in Zurich in 2020.
