Posted on

Teleoperating robots with virtual reality: getting inside a robot’s head

A new VR system from MIT’s Computer Science and Artificial Intelligence Laboratory could make it easy for factory workers to telecommute. (credit: Jason Dorfman, MIT CSAIL)

Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a virtual-reality (VR) system that lets you teleoperate a robot using an Oculus Rift or HTC Vive VR headset.

CSAIL’s “Homunculus Model” system (the classic notion of a small human sitting inside the brain and controlling the actions of the body) embeds you in a VR control room with multiple sensor displays, making it feel like you’re inside the robot’s head. By using gestures, you can control the robot’s matching movements to perform various tasks.

The system can be connected either via a wired local network or via a wireless network connection over the Internet. (The team demonstrated that the system could pilot a robot from hundreds of miles away, testing it on a hotel’s wireless network in Washington, DC to control Baxter at MIT.)

According to CSAIL postdoctoral associate Jeffrey Lipton, lead author on an open-access arXiv paper about the system (presented this week at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) in Vancouver), “By teleoperating robots from home, blue-collar workers would be able to telecommute and benefit from the IT revolution just as white-collars workers do now.”

Jobs for video-gamers too

The researchers imagine that such a system could even help employ jobless video-gamers by “game-ifying” manufacturing positions. (Users with gaming experience had the most ease with the system, the researchers found in tests.)

Homunculus Model system. A Baxter robot (left) is outfitted with a stereo camera rig and various end-effector devices. A virtual control room (user’s view, center), generated on an Oculus Rift CV1 headset (right), allows the user to feel like they are inside Baxter’s head while operating it. Using VR device controllers, including Razer Hydra hand trackers used for inputs (right), users can interact with controls that appear in the virtual space — opening and closing the hand grippers to pick up, move, and retrieve items. A user can plan movements based on the distance between the arm’s location marker and their hand while looking at the live display of the arm. (credit: Jeffrey I. Lipton et al./arXiv).

To make these movements possible, the human’s space is mapped into the virtual space, and the virtual space is then mapped into the robot space to provide a sense of co-location.

The team demonstrated the Homunculus Model system using the Baxter humanoid robot from Rethink Robotics, but the approach could work on other robot platforms, the researchers said.

In tests involving pick and place, assembly, and manufacturing tasks (such as “pick an item and stack it for assembly”) comparing the Homunculus Model system with existing state-of-the-art automated remote-control, CSAIL’s Homunculus Model system had a 100% success rate compared with a 66% success rate for state-of-the-art automated systems. The CSAIL system was also better at grasping objects 95 percent of the time and 57 percent faster at doing tasks.*

“This contribution represents a major milestone in the effort to connect the user with the robot’s space in an intuitive, natural, and effective manner.” says Oussama Khatib, a computer science professor at Stanford University who was not involved in the paper.

The team plans to eventually focus on making the system more scalable, with many users and different types of robots that are compatible with current automation technologies.

* The Homunculus Model system solves a delay problem with existing systems, which use a GPU or CPU, introducing delay. 3D reconstruction from the stereo HD cameras is instead done by the human’s visual cortex, so the user constantly receives visual feedback from the virtual world with minimal latency (delay). This also avoids user fatigue and nausea caused by motion sickness (known as simulator sickness) generated by “unexpected incongruities, such as delays or relative motions, between proprioception and vision [that] can lead to the nausea,” the researchers explain in the paper.


MITCSAIL | Operating Robots with Virtual Reality

Abstract of Baxter’s Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing

Expensive specialized systems have hampered development of telerobotic systems for manufacturing systems. In this paper we demonstrate a telerobotic system which can reduce the cost of such system by leveraging commercial virtual reality(VR) technology and integrating it with existing robotics control software. The system runs on a commercial gaming engine using off the shelf VR hardware. This system can be deployed on multiple network architectures from a wired local network to a wireless network connection over the Internet. The system is based on the homunculus model of mind wherein we embed the user in a virtual reality control room. The control room allows for multiple sensor display, dynamic mapping between the user and robot, does not require the production of duals for the robot, or its environment. The control room is mapped to a space inside the robot to provide a sense of co-location within the robot. We compared our system with state of the art automation algorithms for assembly tasks, showing a 100% success rate for our system compared with a 66% success rate for automated systems. We demonstrate that our system can be used for pick and place, assembly, and manufacturing tasks.

Posted on

A robot with human-like grace and precision

A hybrid hydrostatic transmission and human-safe haptic telepresence robot (credit: Disney Research)

A human-safe lifelike telepresence robot with the delicacy and precision needed to pick up an egg without breaking it or thread a sewing needle has been developed by researchers at Disney Research, the Catholic University of America, and Carnegie Mellon University.

The secret: a hydrostatic transmission that precisely drives robot arms, offering extreme precision with almost no friction or play.

The hybrid transmission design also makes it possible to halve the number of bulky hydraulic lines that a fully hydraulic system would require and allows for making its robotic limbs lighter and smaller, said John P. Whitney, an assistant professor of mechanical and industrial engineering at Northeastern University, who led the development of the transmission while an associate research scientist at Disney Research.

Whitney said a robot joint normally would have two hydraulic cylinders, balanced against each other. But in this latest design, the researchers paired each water-filled cylinder with an air-filled cylinder instead. The pneumatic cylinder serves as a constant force air-spring, providing the necessary preload force and allowing the joint to move in both directions with only half the number of bulky hydraulic lines.

Lifelike interaction with people

The researchers used the new transmission to build a simple humanoid robot with two arms, with stereo cameras mounted in the head, streaming their video signal to an operator wearing a head-mounted display. The arms are coupled to an identical control figure to enable the robot to be used for human-robot interaction research.

“This technology enabled us to build robot arms that are light, fast, and dexterous,” Whitney said. “They have an incredible lifelike nature, offering a combination of small mass, high speed, and precise motion not seen before.”

Robots using this technology are ideally suited for naturally compliant and lifelike interaction with people. When tele-operated, the low friction and lack of play allow the transmission to faithfully transmit contact forces to the operator, providing a high-fidelity remote sense of touch.

Whitney and colleagues will report on the new transmission and the upper body humanoid robot they built with it at the IEEE Conference on Robotics and Automation, ICRA 2016, May 17 in Stockholm, Sweden.


Disney Research | A Hybrid Hydrostatic Transmission and Human Safe Haptic Telepresence Robot

Abstract of A Hybrid Hydrostatic Transmission and Human-Safe Haptic Telepresence Robot

We present a new type of hydrostatic transmission that uses a hybrid air-water configuration, analogous to N+1 cable-tendon transmissions, using N hydraulic lines and 1 pneumatic line for a system with N degrees of freedom (DOFs). The common air-filled line preloads all DOFs in the system, allowing bidirectional operation of every joint. This configuration achieves the high stiffness of a water-filled transmission with half the number of bulky hydraulic lines. We implemented this transmission using pairs of rolling-diaphragm cylinders to form rotary hydraulic actuators, with a new design achieving a 600-percent increase in specific work density per cycle. These actuators were used to build a humanoid robot with two 4-DOF arms, connected via the hydrostatic transmission to an identical master. Stereo cameras mounted on a 2-DOF servo-controlled neck stream live video to the operator’s head-mounted display, which in turn sends the real-time attitude of the operator’s head to the neck servos in the robot. The operator is visually immersed in the robot’s physical workspace, and through the bilateral coupling of the low-impedance hydrostatic transmission, directly feels interaction forces between the robot and external environment. We qualitatively assessed the performance of this system for remote object manipulation and use as a platform to safely study physical human-robot interaction.

Posted on

NYU Holodeck to be model for year 2041 cyberlearning

NYU-X Holodeck (credit: Winslow Burleson and Armanda Lewis)

In an open-access paper in the Journal of Artificial Intelligence Education, Winslow Burleson, PhD, MSE, associate professor, New York University Rory Meyers College of Nursing, suggests that “advanced cyberlearning environments that involve VR and AI innovations are needed to solve society’s “wicked challenges*” — entrenched and seemingly intractable societal problems.

Burleson and and co-author Armanda Lewis imagine such technology in a year 2041 Holodeck, which Burleson’s NYU-X Lab is currently developing in prototype form, in collaboration with colleagues at NYU Courant, Tandon, Steinhardt, and Tisch.

“The “Holodeck” will support a broad range of transdisciplinary collaborations, integrated education, research, and innovation by providing a networked software/hardware infrastructure that can synthesize visual, audio, physical, social, and societal components,” said Burleson.

It’s intended as a model for the future of cyberlearning experience, integrating visual, audio, and physical (haptics, objects, real-time fabrication) components, with shared computation, integrated distributed data, immersive visualization, and social interaction to make possible large-scale synthesis of learning, research, and innovation.

This reminds me of the book Education and Ecstasy, written in 1968 by George B. Leonard, a respected editor for LOOK magazine and, in many respects, a pioneer in what has become the transhumanism movement. That book laid out the justification and promise of advanced educational technology in the classroom for an entire generation. Other writers, such as Harry S. Broudy in the Real World of the Public Schools (1972) followed, arguing that we can not afford “master teachers” in every classroom, but still need to do far better, both then and now.

Today, theories and models of automated planning using computers in complex situations are advanced and “wicked” social simulations can demonstrate the “potholes” in proposed action scenarios. Virtual realties, holodecks, interactive games, robotic and/or AI assistants offer “sandboxes” for learning and for sharing that learning with others. Leonard’s vision, proposed in 1968 for the year 2000, has not yet been realized. However, by 2041, according to these authors, it just might be.

— Warren E. Lacefield, Ph.D. President/CEO Academic Software, Inc.; Associate Professor (retired), Evaluation, Measurement, and Research Program, Department of Educational Leadership, Research, and Technology, Western Michigan University (aka “Asiwel” on KurzweilAI)

Key aspects of the Holodeck: personal stories and interactive experiences that make it a rich environment; open streaming content that make it real and compelling; and contributions that personalize the learning experience. The goal is to create a networked infrastructure and communication environment where “wicked challenges” can be iteratively explored and re-solved, utilizing visual, acoustic, and physical sensory feedback, human dynamics with and social collaboration.

Burleson and Lewis envision that in 2041, learning is unlimited — each individual can create a teacher, team, community, world, galaxy or universe of their own.

* In the late 1960s, urban planners Horst Rittel and Melvin Webber began formulating the concept of “wicked problems” or “wicked challenges” –problems so vexing in the realm of social and organizational planning that they could not be successfully ameliorated with traditional linear, analytical, systems-engineering types of approaches.

These “wicked challenges” are poorly defined, abstruse, and connected to strong moral, political and professional issues.  Some examples might include: “How should we deal with crime and violence in our schools?  “How should we wage the ‘War on Terror’? or “What is good national immigration policy?”

“Wicked problems,” by their very nature, are strongly stakeholder dependent; there is often little consensus even about what the problem is, let alone how to deal with it. And, the challenges themselves are ever shifting sets of inherently complex, interacting issues evolving in a dynamic social context. Often, new forms of “wicked challenges” emerge as a result of trying to understand and treat just one challenge in isolation.

Abstract of Optimists’ Creed: Brave New Cyberlearning, Evolving Utopias (Circa 2041)

This essay imagines the role that artificial intelligence innovations play in the integrated living, learning and research environments of 2041. Here, in 2041, in the context of increasingly complex wicked challenges, whose solutions by their very nature continue to evade even the most capable experts, society and technology have co-evolved to embrace cyberlearning as an essential tool for envisioning and refining utopias–non-existent societies described in considerable detail. Our society appreciates that evolving these utopias is critical to creating and resolving wicked challenges and to better understanding how to create a world in which we are actively “learning to be” – deeply engaged in intrinsically motivating experiences that empower each of us to reach our full potential. Since 2015, Artificial Intelligence in Education (AIED) has transitioned from what was primarily a research endeavour, with educational impact involving millions of user/learners, to serving, now, as a core contributor to democratizing learning (Dewey 2004) and active citizenship for all (billions of learners throughout their lives). An expansive experiential super computing cyberlearning environment, we affectionately call the “Holodeck,” supports transdisciplinary collaboration and integrated education, research, and innovation, providing a networked software/hardware infrastructure that synthesizes visual, audio, physical, social, and societal components. The Holodeck’s large-scale integration of learning, research, and innovation, through real-world problem solving and teaching others what you have learned, effectively creates a global meritocratic network with the potential to resolve society’s wicked challenges while empowering every citizen to realize her or his full potential.

Posted on

Are you ready for soft, morphing, crawling robots with glowing skin displays?

Multi-pixel electroluminescent displays in various states of deformation and illumination (credit: C. Larson et al./Science)

Your future robot or mobile device could have soft, morphable, stretchable “skin” that displays information, according to research by Cornell University engineers. Imagine a health-care robot that displays your blood glucose level and oxygenation, and even your mood — perhaps also your remote physician’s face in 3D.

“When robots become more and more a part of our lives, the ability for them to have an emotional connection with us will be important,” says research team leader Rob Shepherd, an assistant professor of mechanical and aerospace engineering.

Soft robots are currently in use for safe human robot interaction, but they can’t stretch continuously or dynamically display information on their body; and in most cases, can’t sense external and internal stimuli. So the engineers have developed octopus-inspired electroluminescent “skin” that stretches to more than six times its original size, and can also change shape and color.

An undulating gait produced by pressurizing the chambers in sequence along the length of the crawler (credit: C. Larson et al./Science)

The new technology uses a “hyper-elastic light-emitting capacitor” (HLEC), consisting of layers of transparent hydrogel electrodes sandwiching an insulating elastomer (a polymer with viscoelasticity, meaning it has both viscosity and elasticity) sheet.

The elastomer changes luminance and capacitance (the ability to store an electrical charge) when stretched, rolled, and otherwise deformed.

The HLEC skin also endows soft robots with the ability to sense their actuated state and environment and communicate optically — and (for small robots) even crawl.

The engineers created a prototype crawling soft robot, using three of the six-layer HLEC panels bound together. The top four layers made up the illuminated skin and the bottom two served as pneumatic actuators. The chambers were alternately inflated and deflated; the resulting curvature created an undulating, walking motion.

It could also make for a fun pet, we’re guessing.

The team’s research was published in the March 3 online edition of the journal Science. It was supported by a grant from the Army Research Office, a 2015 award from the Air Force Office of Scientific Research, and two grants from the National Science Foundation.

Cornell University | Electroluminescent Skin Demonstration

Abstract of Highly stretchable electroluminescent skin for optical signaling and tactile sensing

Cephalopods such as octopuses have a combination of a stretchable skin and color-tuning organs to control both posture and color for visual communication and disguise. We present an electroluminescent material that is capable of large uniaxial stretching and surface area changes while actively emitting light. Layers of transparent hydrogel electrodes sandwich a ZnS phosphor-doped dielectric elastomer layer, creating thin rubber sheets that change illuminance and capacitance under deformation. Arrays of individually controllable pixels in thin rubber sheets were fabricated using replica molding and were subjected to stretching, folding, and rolling to demonstrate their use as stretchable displays. These sheets were then integrated into the skin of a soft robot, providing it with dynamic coloration and sensory feedback from external and internal stimuli.

Posted on

Google Glass could bring toxicology specialists to remote emergency rooms

(credit: Google)

Researchers at the University of Massachusetts Medical School have found that Google Glass — presumably the Enterprise Edition — could effectively extend bedside toxicology consults to distant health care facilities such as community and rural hospitals to diagnose and manage poisoned patients, according to a paper in the Journal of Medical Toxicology.

“In the present era of value-based care, a toxicology service using hands-free devices, such as Google Glass, could conceivably expand its coverage area and enhance patient care, while potentially decreasing overall treatment costs,” said Peter R. Chai, MD, toxicology fellow at UMass Medical School. “Our work shows that the data transmitted by Google Glass can be used to supplement traditional telephone consults, validate bedside physical exams, and diagnose and manage patients.”

Traditional telemedicine devices usually consist of large desktop or laptop computers affixed to a big cart that has to be rolled from exam room to exam room. “Glass is positioned perfectly as an emergency medicine telemedical device. Its small, hands free and portable, so you can bring it right to the bedside and have a real-time specialist with you when you need one,” he said.

In the study, emergency medicine residents at UMass Memorial Medical Center performed 18 toxicology consults with Google Glass. ER physicians wearing Google Glass evaluated the patients at bedside while a secure video feed was sent to the toxicology supervising consultant. The supervising consultant then guided the resident through text messages displayed on the Glass. Consultants also obtained static photos of medication bottles, electrocardiograms (EKG) and other pertinent information at the discretion of the supervisor.

As a result of using Google Glass, consulting toxicologists reported being more confident in diagnosing specific toxidromes. Additional data collected showed that the use of Google Glass also changed management of patient care in more than half of the cases seen. Specifically, six of those patients received antidotes they otherwise would not have. Overall, 89 percent of the cases seen with Glass were considered successful by the consulting toxicologist.

Google currently lists several companies involved in the medical field as Glass At Work partners, such as Advanced Medical Applications, which specializes in “solutions in telemedicine, live-surgery demonstrations, and remote medical training.”

According to 9to5Google sources, the Google Glass Enterprise Edition will feature “a robust hinge mechanism that allows the computer and battery modules to fold down like a regular pair of glasses, and a hinge for folding down the left side of the band as well.” It also “includes a larger prism display for a better viewing experience, an Intel Atom processor that brings better performance, moderately improved battery life, and better heat management.”

Abstract of The Feasibility and Acceptability of Google Glass for Teletoxicology Consults

Teletoxicology offers the potential for toxicologists to assist in providing medical care at remote locations, via remote, interactive augmented audiovisual technology. This study examined the feasibility of using Google Glass, a head-mounted device that incorporates a webcam, viewing prism, and wireless connectivity, to assess the poisoned patient by a medical toxicology consult staff. Emergency medicine residents (resident toxicology consultants) rotating on the toxicology service wore Glass during bedside evaluation of poisoned patients; Glass transmitted real-time video of patients’ physical examination findings to toxicology fellows and attendings (supervisory consultants), who reviewed these findings. We evaluated the usability (e.g., quality of connectivity and video feeds) of Glass by supervisory consultants, as well as attitudes towards use of Glass. Resident toxicology consultants and supervisory consultants completed 18 consults through Glass. Toxicologists viewing the video stream found the quality of audio and visual transmission usable in 89 % of cases. Toxicologists reported their management of the patient changed after viewing the patient through Glass in 56 % of cases. Based on findings obtained through Glass, toxicologists recommended specific antidotes in six cases. Head-mounted devices like Google Glass may be effective tools for real-time teletoxicology consultation.