Posted on

VR glove powered by soft robotics provides missing sense of touch

Prototype of haptic VR glove, using soft robotic “muscles” to provide realistic tactile feedback for VR experiences (credit: Jacobs School of Engineering/UC San Diego)

Engineers at UC San Diego have designed a light, flexible glove with soft robotic muscles that provide realistic tactile feedback for virtual reality (VR) experiences.

Currently, VR tactile-feedback user interfaces are bulky, uncomfortable to wear and clumsy, and they simply vibrate when a user touches a virtual surface or object.

“This is a first prototype, but it is surprisingly effective,” said Michael Tolley, a mechanical engineering professor at the Jacobs School of Engineering at UC San Diego and a senior author of a paper presented at the Electronic Imaging, Engineering Reality for Virtual Reality conference in Burlingame, California and published May 31, 2017 in Advanced Engineering Materials.

The key soft-robotic component of the new glove is a version of the “McKibben muscle” (a pneumatic, or air-based, actuator invented in 1950s by the physician Joseph L. McKibben for use in prosthetic limbs), using soft latex chambers covered with braided fibers. To apply tactile feedback when the user moves their fingers, the muscles respond like springs. The board controls the muscles by inflating and deflating them.*

Prototype haptic VR glove system. A computer generates an image of a virtual world (in this case, a piano keyboard with a river and trees in the background) that it sends to the VR device (such as an Oculus Rift). A Leap Motion depth-camera (on the table) detects the position and movement of the user’s hands and sends data to a computer. It sends an image of the user’s hands superimposed over the keyboard (in the demo case) to the VR display and to a custom fluidic control board. The board then feeds back a signal to soft robotic components in the glove to individually inflate or deflate fingers, mimicking the user’s applied forces.

The engineers conducted an informal pilot study of 15 users, including two VR interface experts. The demo allowed them to play the piano in VR. They all agreed that the gloves increased the immersive experience, which they described as “mesmerizing” and “amazing.”

VR headset image of a piano, showing user’s finger actions (credit: Jacobs School of Engineering/UC San Diego)

The engineers say they’re working on making the glove cheaper, less bulky, and more portable. They would also like to bypass the Leap Motion device altogether to make the system more self-contained and compact. “Our final goal is to create a device that provides a richer experience in VR,” Tolley said. “But you could imagine it being used for surgery and video games, among other applications.”

* The researchers 3D-printed a mold to make the gloves’ soft exoskeleton. This will make the devices easier to manufacture and suitable for mass production, they said. Researchers used silicone rubber for the exoskeleton, with Velcro straps embedded at the joints.


JacobsSchoolNews | A glove powered by soft robotics to interact with virtual reality environments

Abstract of Soft Robotics: Review of Fluid-Driven Intrinsically Soft Devices; Manufacturing, Sensing, Control, and Applications in Human-Robot Interaction

The emerging field of soft robotics makes use of many classes of materials including metals, low glass transition temperature (Tg) plastics, and high Tg elastomers. Dependent on the specific design, all of these materials may result in extrinsically soft robots. Organic elastomers, however, have elastic moduli ranging from tens of megapascals down to kilopascals; robots composed of such materials are intrinsically soft − they are always compliant independent of their shape. This class of soft machines has been used to reduce control complexity and manufacturing cost of robots, while enabling sophisticated and novel functionalities often in direct contact with humans. This review focuses on a particular type of intrinsically soft, elastomeric robot − those powered via fluidic pressurization.

Posted on

Common antioxidant could slow symptoms of aging in human skin

These cross-section images show three-dimensional human skin models made of living skin cells. Untreated model skin (left panel) shows a thinner dermis layer (black arrow) compared with model skin treated with the antioxidant methylene blue (right panel). A new study suggests that methylene blue could slow or reverse dermal thinning (a sign of aging) and a number of other symptoms of aging in human skin. (credit: Zheng-Mei Xiong/University of Maryland)

University of Maryland (UMD) researchers have found evidence that a common, inexpensive, and safe antioxidant chemical called methylene blue could slow the aging of human skin, based on tests in cultured human skin cells and simulated skin tissue.

“The effects we are seeing are not temporary. Methylene blue appears to make fundamental, long-term changes to skin cells,” said Kan Cao, senior author on the study and an associate professor of cell biology and molecular genetics at UMD.

The researchers tested methylene blue for four weeks in skin cells from healthy middle-aged donors, as well as those diagnosed with progeria — a rare genetic disease that mimics the normal aging process at an accelerated rate. The researchers also tested three other known antioxidants: N-Acetyl-L-Cysteine (NAC), MitoQ and MitoTEMPO (mTEM).

In these experiments, methylene blue outperformed the other three antioxidants, improving several age-related symptoms in cells from both healthy donors and progeria patients. The skin cells (fibroblasts, the cells that produce the structural protein collagen) experienced a decrease in damaging molecules known as reactive oxygen species (ROS), a reduced rate of cell death, and an increase in the rate of cell division throughout the four-week treatment.

Improvements in skin cells from older donors (>80 years old)

Next, Cao and her colleagues tested methylene blue in fibroblasts from older donors (>80 years old), again for a period of four weeks. At the end of the treatment, the cells from older donors had experienced a range of improvements, including decreased expression of two genes commonly used as indicators of cellular aging: senescence-associated beta-galactosidase and p16.

Schematic illustrations of top (left panel) and side (right panel) views of the engineered 3D skin tissue cultured on a microporous membrane insert, used for experiments and skin-irritation tests (credit: Zheng-Mei Xiong et al./Scientific Reports)

The researchers then used simulated human skin to perform several more experiments. This simulated skin — a three-dimensional model made of living skin cells — includes all the major layers and structures of skin tissue, with the exception of hair follicles and sweat glands. The model skin could also be used in skin irritation tests required by the Food and Drug Administration for the approval of new cosmetic products, Cao said.

“This system allowed us to test a range of aging symptoms that we can’t replicate in cultured cells alone,” Cao said. “Most surprisingly, we saw that model skin treated with methylene blue retained more water and increased in thickness—both of which are features typical of younger skin.”

Formulating cosmetics

The researchers also used the model skin to test the safety of cosmetic creams with methylene blue added. The results suggest that methylene blue causes little to no irritation, even at high concentrations. Encouraged by these results, Cao and colleagues hope to develop safe and effective ways for consumers to benefit from the properties of methylene blue.

“We have already begun formulating cosmetics that contain methylene blue. Now we are looking to translate this into marketable products,” Cao said. “Perhaps down the road we can customize the system with bioprinting, such that we might be able to use a patient’s own cells to provide a tailor-made testing platform specific to their needs.”

The study was published online in the Nature journal Scientific Reports on May 30, 2017.

This research was supported by the Maryland Innovation Initiative.

Abstract of Anti-Aging Potentials of Methylene Blue for Human Skin Longevity

Oxidative stress is the major cause of skin aging that includes wrinkles, pigmentation, and weakened wound healing ability. Application of antioxidants in skin care is well accepted as an effective approach to delay the skin aging process. Methylene blue (MB), a traditional mitochondrial-targeting antioxidant, showed a potent ROS scavenging efficacy in cultured human skin fibroblasts derived from healthy donors and from patients with progeria, a genetic premature aging disease. In comparison with other widely used general and mitochondrial-targeting antioxidants, we found that MB was more effective in stimulating skin fibroblast proliferation and delaying cellular senescence. The skin irritation test, performed on an in vitro reconstructed 3D human skin model, indicated that MB was safe for long-term use, and did not cause irritation even at high concentrations. Application of MB to this 3D skin model further demonstrated that MB improved skin viability, promoted wound healing and increased skin hydration and dermis thickness. Gene expression analysis showed that MB treatment altered the expression of a subset of extracellular matrix proteins in the skin, including upregulation of elastin and collagen 2A1, two essential components for healthy skin. Altogether, our study suggests that MB has a great potential for skin care.

Posted on

New antibiotic could eliminate the global threat of antibiotic-resistant infections

Modified vancomycin antibiotic (credit: Akinori Okano et al./PNAS)

Scientists at The Scripps Research Institute (TSRI) have discovered a way to structurally modify the antibiotic called vancomycin to make an already-powerful version of the antibiotic even more potent — an advance that could eliminate the threat of antibiotic-resistant infections for years to come.

“Doctors could use this modified form of vancomycin without fear of resistance emerging,” said Dale Boger, co-chair of TSRI’s Department of Chemistry, whose team announced the finding Monday (May 29, 2016) in the journal Proceedings of the National Academy of Sciences.

“The death of a hospitalized patient in Reno Nevada for whom no available antibiotics worked highlights what World Health Organization and other public-health experts have been warning: antibiotic resistance is a serious threat and has gone global,” KurzweilAI reported in January 2017. The new finding promises to lead to a solution.

First antibiotic to have three independent mechanisms of action

Vancomycin  has been prescribed by doctors for 60 years, and bacteria are only now developing resistance to it, according to Boger, who called vancomycin “magical” for its proven strength against infections. Previous studies by Boger and his colleagues at TSRI had shown that it is possible to add two modifications to vancomycin to make it even more potent. “With these modifications, you need less of the drug to have the same effect,” Boger said.

The new study shows that scientists can now make a third modification that interferes with a bacterium’s cell wall in a new way, with promising results. Combined with the previous modifications, this alteration gives vancomycin a 1,000-fold increase in activity, meaning doctors would need to use less of the antibiotic to fight infection.

The discovery makes this version of vancomycin the first antibiotic to have three independent mechanisms of action. “This increases the durability of this antibiotic,” said Boger. “Organisms just can’t simultaneously work to find a way around three independent mechanisms of action. Even if they found a solution to one of those, the organisms would still be killed by the other two.”

Tested against Enterococci bacteria, the new version of vancomycin killed both vancomycin-resistant Enterococci and the original forms of Enterococci. The next step in this research is to design a way to synthesize the modified vancomycin using fewer steps in the lab; the current method takes 30 steps.

The study was supported by the National Institutes of Health.

Posted on

Are you ready for pop-up, shape-shifting food? Just add water.

Fun with food: These pasta shapes were generated by immersing a 2D flat gelatin film into water. (credit: Michael Indresano Photography)

Researchers at MIT’s Tangible Media Group are exploring ways to make your dining experience interactive and fun, with food that can transform its shape by just adding water.

Think of it as edible origami or culinary performance art — flat sheets of gelatin and starch that instantly sprout into three-dimensional structures, such as macaroni and rotini, or the shape of a flower.

But the researchers suggest it’s also a practical way to reduce food-shipping costs. Edible films could be stacked together, IKEA-style, and shipped to consumers, then morph into their final shape later when immersed in water.

“We did some simple calculations, such as for macaroni pasta, and even if you pack it perfectly, you still will end up with 67 percent of the volume as air,” says Wen Wang, a co-author on the paper and a former graduate student and research scientist in MIT’s Media Lab. “We thought maybe in the future our shape-changing food could be packed flat and save space.”

Programmable pasta, anyone?

At MIT, Wang and associates had been investigating the response of various materials to moisture. They started playing around with gelatin (as in Jello), a substance that naturally expands when it absorbs water. Gelatin can expand to varying degrees depending on its density — a characteristic that the team exploited in creating their shape-transforming structures.

They created a flat, two-layer film made from gelatin of two different densities. In theory, the top layer was more densely packed, so it should be able to absorb more water than the bottom layer. Sure enough, when they immersed the entire structure in water, the top layer curled over the bottom layer, forming a slowly rising arch — creative pasta.*

Culinary performance art by MIT  researchers. (left) Phytoplankton pasta salad with heirloom tomatoes and wild Sorrel. (right) Flowering pasta with west-coast foraged mushrooms
and fermented burgundy truffle. (credit: Michael Indresano Photography)

To see how their designs might be implemented in a professional kitchen, the researchers showed their engineered edibles to Matthew Delisle, the head chef of high-end Boston restaurant L’Espalier. They jointly designed two culinary creations: transparent discs of gelatin flavored with plankton and squid ink, that instantly wrap around small beads of caviar; and long fettuccini-like strips, made from two gelatins that melt at different temperatures, causing the noodles to spontaneously divide when hot broth melts away certain sections. “They had great texture and tasted pretty good,” Yao says.

DIY food 

The researchers used a laboratory 3-D printer to pattern cellulose onto films of gelatin. But they suggest users can reproduce similar effects with more common techniques such as “screen printing” in an open-access paper presented at the Association for Computing Machinery’s 2017 Computer-Human Interaction Conference on Human Factors in Computing Systems (CHI 2017).

They envision that their “online software can provide design instructions, and a startup company can ship the materials to your home,” Yao says.

This research was funded, in part, by the MIT Media Lab and Food + Future, a startup accelerator sponsored by Target Corporation, IDEO, and Intel.

* The team recorded the cellulose patterns and the dimensions of all of the structures they were able to produce, and also tested mechanical properties such as toughness, organizing all this data into a database. Co-authors Zhang and Cheng then built computational models of the material’s transformations, which they used to design an online interface for users to design their own edible, shape-transforming structures.“We did many lab tests and collected a database, within which you can pick different shapes, with fabrication instructions,” Wang says. “Reversibly, you can also select a basic pattern from the database and adjust the distribution or thickness, and can see how the final transformation will look.”


Tangible Media Group | Transformative Appetite

Abstract of Transformative Appetite: Shape-Changing Food Transforms from 2D to 3D by Water Interaction through Cooking

We developed a concept of transformative appetite, where edible 2D films made of common food materials (protein, cellulose or starch) can transform into 3D food during cooking. This transformation process is triggered by water adsorption, and it is strongly compatible with the ‘flat packaging’ concept for substantially reducing shipping costs and storage space. To develop these transformable foods, we performed material-based design, established a hybrid fabrication strategy, and conducted performance simulation. Users can customize food shape transformations through a pre-defined simulation platform, and then fabricate these designed patterns using additive manufacturing. Three application techniques are provided – 2D-to-3D folding, hydration-induced wrapping, and temperature-induced self-fragmentation, to present the shape, texture, and interaction with food materials. Based on this concept, several dishes were created in the kitchen, to demonstrate the futuristic dining experience through materials-based interaction design.

Posted on

Alpha Go defeats world’s top Go player. What’s next?

What does the research team behind AlphaGo do next after winning the three-game match Saturday (May 27) against Ke Jie — the world’s top Go player — at the Future of Go Summit in Wuzhen, China?

“Throw their energy into the next set of grand challenges, developing advanced general algorithms that could one day help scientists as they tackle some of our most complex problems, such as finding new cures for diseases, dramatically reducing energy consumption, or inventing revolutionary new materials,” says DeepMind Technologies CEO Demis Hassabis.

Academic paper, Go teaching tool

But it’s “not the end of our work with the Go community,” he adds. “We plan to publish one final academic paper later this year that will detail the extensive set of improvements we made to the algorithms’ efficiency and potential to be generalised across a broader set of problems.”

Already in the works (with Jie’s collaboration): a teaching tool that “will show AlphaGo’s analysis of Go positions, providing an insight into how the program thinks, and hopefully giving all players and fans the opportunity to see the game through the lens of AlphaGo.”

Ke Jie plays the final match (credit: DeepMind)

DeepMind is also “publishing a special set of 50 AlphaGo vs AlphaGo games, played at full-length time controls, which we believe contain many new and interesting ideas and strategies.”


Deep Mind | The Future of Go Summit, Match Three: Ke Jie & AlphaGo


Deep Mind | Exploring the mysteries of Go with AlphaGo and China’s top players


DeepMind | Demis Hassabis on AlphaGo: its legacy and the ‘Future of Go Summit’ in Wuzhen, China

Posted on

‘Wearable’ PET brain scanner enables studies of moving patients

Julie Brefczynski-Lewis, a neuroscientist at West Virginia University, places a helmet-like PET scanner on a research subject. The mobile scanner enables studies of human interaction, movement disorders, and more. (credit: West Virginia University)

Two scientists have developed a miniaturized positron emission tomography (PET) brain scanner that can be “worn” like a helmet.

The new Ambulatory Microdose Positron Emission Tomography (AMPET) scanner allows research subjects to stand and move around as the device scans, instead of having to lie completely still and be administered anesthesia — making it impossible to find associations between movement and brain activity.

Conventional positron emission tomography (PET) scanners immobilize patients (credit: Jens Maus/CC)

The AMPET scanner was developed by Julie Brefczynski-Lewis, a neuroscientist at West Virginia University (WVU), and Stan Majewski, a physicist at WVU and now at the University of Virginia. It could make possible new psychological and clinical studies on how the brain functions when affected by diseases from epilepsy to addiction, and during ordinary and dysfunctional social interactions.

Helmet support prototype with weighted helmet, allowing for freedom of movement. The counterbalance currently supports up to 10 kg but can be upgraded. Digitizing electronics will be mounted to the support above the patient. (credit: Samantha Melroy et al./Sensors)

Because AMPET sits so close to the brain, it can also “catch” more of the photons stemming from the radiotracers used in PET than larger scanners can. That means researchers can administer a lower dose of radioactive material and still get a good biological snapshot. Catching more signals also allows AMPET to create higher resolution images than regular PET.

The AMPET idea was sparked by the Rat Conscious Animal PET (RatCAP) scanner for studying rats at the U.S. Department of Energy’s (DOE) Brookhaven National Laboratory.** The scanner is a 250-gram ring that fits around the head of a rat, suspended by springs to support its weight and let the rat scurry about as the device scans. (credit: Brookhaven Lab)

The researchers plan to build a laboratory-ready version next.

Seeing more deeply into the brain

A patient or animal about to undergo a PET scan is injected with a low dose of a radiotracer — a radioactive form of a molecule that is regularly used in the body. These molecules emit anti-matter particles called positrons, which then manage to only travel a tiny distance through the body. As soon as one of these positrons meets an electron in biological tissue, the pair annihilates and converts their mass to energy. This energy takes the form of two high-energy light rays, called gamma photons, that shoot off in opposite directions. PET machines detect these photons and track their paths backward to their point of origin — the tracer molecule. By measuring levels of the tracer, for instance, doctors can map areas of high metabolic activity. Mapping of different tracers provides insight into different aspects of a patient’s health. (credit: Brookhaven Lab)

PET scans allow researchers to see farther into the body than other imaging tools. This lets AMPET reach deep neural structures while the research subjects are upright and moving. “A lot of the important things that are going on with emotion, memory, and behavior are way deep in the center of the brain: the basal ganglia, hippocampus, amygdala,” Brefczynski-Lewis notes.

“Currently we are doing tests to validate the use of virtual reality environments in future experiments,” she said. In this virtual reality, volunteers would read from a script designed to make the subject angry, for example, as his or her brain is scanned.

In the medical sphere, the scanning helmet could help explain what happens during drug treatments. Or it could shed light on movement disorders such as epilepsy, and watch what happens in the brain during a seizure; or study the sub-population of Parkinson’s patients who have great difficulty walking, but can ride a bicycle .

The RatCAP project at Brookhaven was funded by the DOE Office of Science. RHIC is a DOE Office of Science User Facility for nuclear physics research. Brookhaven Lab physicists use technology similar to PET scanners at the Relativistic Heavy Ion Collider (RHIC), where they must track the particles that fly out of near-light speed collisions of charged nuclei. PET research at the Lab dates back to the early 1960s and includes the creation of the first single-plane scanner as well as various tracer molecules.

Abstract of Development and Design of Next-Generation Head-Mounted Ambulatory Microdose Positron-Emission Tomography (AM-PET) System

Several applications exist for a whole brain positron-emission tomography (PET) brain imager designed as a portable unit that can be worn on a patient’s head. Enabled by improvements in detector technology, a lightweight, high performance device would allow PET brain imaging in different environments and during behavioral tasks. Such a wearable system that allows the subjects to move their heads and walk—the Ambulatory Microdose PET (AM-PET)—is currently under development. This imager will be helpful for testing subjects performing selected activities such as gestures, virtual reality activities and walking. The need for this type of lightweight mobile device has led to the construction of a proof of concept portable head-worn unit that uses twelve silicon photomultiplier (SiPM) PET module sensors built into a small ring which fits around the head. This paper is focused on the engineering design of mechanical support aspects of the AM-PET project, both of the current device as well as of the coming next-generation devices. The goal of this work is to optimize design of the scanner and its mechanics to improve comfort for the subject by reducing the effect of weight, and to enable diversification of its applications amongst different research activities.

Posted on

3D-printed ‘bionic skin’ could give robots and prosthetics the sense of touch

Schematic of a new kind of 3D printer that can print touch sensors directly on a model hand. (credit: Shuang-Zhuang Guo and Michael McAlpine/Advanced Materials )

Engineering researchers at the University of Minnesota have developed a process for 3D-printing stretchable, flexible, and sensitive electronic sensory devices that could give robots or prosthetic hands — or even real skin — the ability to mechanically sense their environment.

One major use would be to give surgeons the ability to feel during minimally invasive surgeries instead of using cameras, or to increase the sensitivity of surgical robots. The process could also make it easier for robots to walk and interact with their environment.

Printing electronics directly on human skin could be used for pulse monitoring, energy harvesting (of movements), detection of finger motions (on a keyboard or other devices), or chemical sensing (for example, by soldiers in the field to detect dangerous chemicals or explosives). Or imagine a future computer mouse built into your fingertip, with haptic touch on any surface.

“While we haven’t printed on human skin yet, we were able to print on the curved surface of a model hand using our technique,” said Michael McAlpine, a University of Minnesota mechanical engineering associate professor and lead researcher on the study.* “We also interfaced a printed device with the skin and were surprised that the device was so sensitive that it could detect your pulse in real time.”

The researchers also visualize use in “bionic organs.”

A unique skin-compatible 3D-printing process

(left) Schematic of the tactile sensor. (center) Top view. (right) Optical image showing the conformally printed 3D tactile sensor on a fingertip. Scale bar = 4 mm. (credit: Shuang-Zhuang Guo et al./Advanced Materials)

McAlpine and his team made the sensing fabric with a one-of-a kind 3D printer they built in the lab. The multifunctional printer has four nozzles to print the various specialized “inks” that make up the layers of the device — a base layer of silicone**, top and bottom electrodes made of a silver-based piezoresistive conducting ink, a coil-shaped pressure sensor, and a supporting layer that holds the top layer in place while it sets (later washed away in the final manufacturing process).

Surprisingly, all of the layers of “inks” used in the flexible sensors can set at room temperature. Conventional 3D printing using liquid plastic is too hot and too rigid to use on the skin. The sensors can stretch up to three times their original size.

The researchers say the next step is to move toward semiconductor inks and printing on a real surface. “The manufacturing is built right into the process, so it is ready to go now,” McAlpine said.

The research was published online in the journal Advanced Materials. It was funded by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health.

* McAlpine integrated electronics and novel 3D-printed nanomaterials to create a “bionic ear” in 2013.

** The silicone rubber has a low modulus of elasticity of 150 kPa, similar to that of skin, and lower hardness (Shore A 10) than that of human skin, according to the Advanced Materials paper.


College of Science and Engineering, UMN | 3D Printed Stretchable Tactile Sensors

Abstract of 3D Printed Stretchable Tactile Sensors

The development of methods for the 3D printing of multifunctional devices could impact areas ranging from wearable electronics and energy harvesting devices to smart prosthetics and human–machine interfaces. Recently, the development of stretchable electronic devices has accelerated, concomitant with advances in functional materials and fabrication processes. In particular, novel strategies have been developed to enable the intimate biointegration of wearable electronic devices with human skin in ways that bypass the mechanical and thermal restrictions of traditional microfabrication technologies. Here, a multimaterial, multiscale, and multifunctional 3D printing approach is employed to fabricate 3D tactile sensors under ambient conditions conformally onto freeform surfaces. The customized sensor is demonstrated with the capabilities of detecting and differentiating human movements, including pulse monitoring and finger motions. The custom 3D printing of functional materials and devices opens new routes for the biointegration of various sensors in wearable electronics systems, and toward advanced bionic skin applications.

Posted on

How Google’s ‘smart reply’ is getting smarter

(credit: Google Research)

Last week, KurzweilAI reported that Google is rolling out an enhanced version of its “smart reply” machine-learning email software to “over 1 billion Android and iOS users of Gmail” — quoting Google CEO Sundar Pichai.

We noted that the new smart-reply version is now able to handle challenging sentences like “That interesting person at the cafe we like gave me a glance,” as Google research scientist Brian Strope and engineering director Ray Kurzweil noted in a Google Research blog post.

But “given enough examples of language, a machine learning approach can discover many of these subtle distinctions,” they wrote.

How does it work? “The content of language is deeply hierarchical, reflected in the structure of language itself, going from letters to words to phrases to sentences to paragraphs to sections to chapters to books to authors to libraries, etc.,” they explained.

So a hierarchical approach to learning “is well suited to the hierarchical nature of language. We have found that this approach works well for suggesting possible responses to emails. We use a hierarchy of modules, each of which considers features that correspond to sequences at different temporal scales, similar to how we understand speech and language.”*

Simplfying communication

“With Smart Reply, Google is assuming users want to offload the burdensome task of communicating with one another to our more efficient counterparts,” says Wired writer Liz Stinson.

“It’s not wrong. The company says the machine-generated replies already account for 12 percent of emails sent; expect that number to boom once everyone with the Gmail app can send one-tap responses.

“In the short term, that might mean more stilted conversations in your inbox. In the long term, the growing number of people who use these canned responses is only going to benefit Google, whose AI grows smarter with every email sent.”

Another challenge is that our emails, particularly from mobile devices, “tend to be riddled with idioms [such as urban lingo] that make no actual sense,” suggests Washington Post writer Hayley Tsukayama. “Things change depending on context: Something ‘wicked’ could be good or very bad, for example. Not to mention, sarcasm is a thing.

“Which is all to warn you that you may still get a wildly random and even potentially inappropriate suggestion — I once got an ‘Oh no!’ suggestion to a friend’s self-deprecating pregnancy announcement, for example. If the email only calls for a one- or two-sentence response, you’ll probably find Smart Reply useful. If it requires any nuance, though, it’s still best to use your own human judgment.”

* The initial release of Smart Reply encoded input emails word-by-word with a long-short-term-memory (LSTM) recurrent neural network, and then decoded potential replies with yet another word-level LSTM. While this type of modeling is very effective in many contexts, even with Google infrastructure, it’s an approach that requires substantial computation resources. Instead of working word-by-word, we found an effective and highly efficient path by processing the problem more all-at-once, by comparing a simple hierarchy of vector representations of multiple features corresponding to longer time spans. — Brian Strope and Ray Kurzweil, Google Research Blog.

Posted on

When AI improves human performance instead of taking over

The game results show that placing slightly “noisy” bots in a central location (high-degree nodes) improves human coordination by reducing same-color neighbor nodes (the goal of the game). Square nodes show the bots and round nodes show human players; thick red lines show color conflicts, which are reduced with bot participation (right). (credit: Hirokazu Shirado and Nicholas A. Christakis/Nature)

It’s not about artificial intelligence (AI) taking over — it’s about AI improving human performance, a new study by Yale University researchers has shown.

“Much of the current conversation about artificial intelligence has to do with whether AI is a substitute for human beings. We believe the conversation should be about AI as a complement to human beings,” said Nicholas Christakis, Yale University co-director of the Yale Institute for Network Science (YINS) and senior author of a study by Yale Institute for Network Science.*

AI doesn’t even have to be super-sophisticated to make a difference in people’s lives; even “dumb AI” can help human groups, based on the study, which appears in the May 18, 2017 edition of the journal Nature.

How bots can boost human performance

In a series of experiments using teams of human players and autonomous software agents (“bots”), the bots boosted the performance of human groups and the individual players, the researchers found.

The experiment design involved an online color-coordination game that required groups of people to coordinate their actions for a collective goal. The collective goal was for every node to have a color different than all of its neighbor nodes. The subjects were paid a US$2 show-up fee and a declining bonus of up to US$3 depending on the speed of reaching a global solution to the coordination problem (in which every player in a group had chosen a different color from their connected neighbors). When they did not reach a global solution within 5 min, the game was stopped and the subjects earned no bonus.

The human players also interacted with anonymous bots that were programmed with three levels of behavioral randomness — meaning the AI bots sometimes deliberately made mistakes (introduced “noise”). In addition, sometimes the bots were placed in different parts of the social network to try different strategies.

The result: The bots reduced the median time for groups to solve problems by 55.6%. The experiment also showed a cascade effect: People whose performance improved when working with the bots then influenced other human players to raise their game. More than 4,000 people participated in the experiment, which used Yale-developed software called breadboard.

The findings have implications for a variety of situations in which people interact with AI technology, according to the researchers. Examples include human drivers who share roadways with autonomous cars and operations in which human soldiers work in tandem with AI.

“There are many ways in which the future is going to be like this,” Christakis said. “The bots can help humans to help themselves.”

Practical business AI tools

One example: Salesforce CEO Marc Benioff uses a bot called Einstein to help him run his company, Business Intelligence reported Thursday (May 18, 2017).

 

“Powered by advanced machine learning, deep learning, predictive analytics, natural language processing and smart data discovery, Einstein’s models will be automatically customised for every single customer,” according to the Salesforce blog. “It will learn, self-tune and get smarter with every interaction and additional piece of data. And most importantly, Einstein’s intelligence will be embedded within the context of business, automatically discovering relevant insights, predicting future behavior, proactively recommending best next actions and even automating tasks.”

Benioff says he also uses a version called Einstein Guidance for forecasting and modeling. It even helps end internal politics at executive meetings, calling out under-performing executives.

“AI is the next platform. All future apps for all companies will be built on AI,” Benioff predicts.

* Christakis is a professor of sociology, ecology & evolutionary biology, biomedical engineering, and medicine at Yale. Grants from the Robert Wood Johnson Foundation and the National Institute of Social Sciences supported the research.

Abstract of Locally noisy autonomous agents improve global human coordination in network experiments

Coordination in groups faces a sub-optimization problem and theory suggests that some randomness may help to achieve global optima. Here we performed experiments involving a networked colour coordination game in which groups of humans interacted with autonomous software agents (known as bots). Subjects (n = 4,000) were embedded in networks (n = 230) of 20 nodes, to which we sometimes added 3 bots. The bots were programmed with varying levels of behavioural randomness and different geodesic locations. We show that bots acting with small levels of random noise and placed in central locations meaningfully improve the collective performance of human groups, accelerating the median solution time by 55.6%. This is especially the case when the coordination problem is hard. Behavioural randomness worked not only by making the task of humans to whom the bots were connected easier, but also by affecting the gameplay of the humans among themselves and hence creating further cascades of benefit in global coordination in these heterogeneous systems.

Posted on

Princeton/Adobe technology will let you edit voices like text

Technology developed by Princeton University computer scientists may do for audio recordings of the human voice what word processing software did for the written word and Adobe Photoshop did for images.

“VoCo” software, still in the research stage, makes it easy to add or replace a word in an audio recording of a human voice by simply editing a text transcript of the recording. New words are automatically synthesized in the speaker’s voice — even if they don’t appear anywhere else in the recording.

The system uses a sophisticated algorithm to learn and recreate the sound of a particular voice. It could one day make editing podcasts and narration in videos much easier, or in the future, create personalized robotic voices that sound natural, according to co-developer Adam Finkelstein, a professor of computer science at Princeton. Or people who have lost their voices due to injury or disease might be able to recreate their voices through a robotic system, but one that sounds natural.

An earlier version of VoCo was announced in November 2016. A paper describing the current VoCo development will be published in the July issue of the journal Transactions on Graphics (an open-access preprint is available).

How it works (technical description)

VoCo allows people to edit audio recordings with the ease of changing words on a computer screen. The system inserts new words in the same voice as the rest of the recording. (credit: Professor Adam Finkelstein)

VoCo’s user interface looks similar to other audio editing software such as the podcast editing program Audacity, with a waveform of the audio track and cut, copy and paste tools for editing. But VoCo also augments the waveform with a text transcript of the track and allows the user to replace or insert new words that don’t already exist in the track by simply typing in the transcript. When the user types the new word, VoCo updates the audio track, automatically synthesizing the new word by stitching together snippets of audio from elsewhere in the narration.

VoCo is is based on an optimization algorithm that searches the voice recording and chooses the best possible combinations of phonemes (partial word sounds) to build new words in the user’s voice. To do this, it needs to find the individual phonemes and sequences of them that stitch together without abrupt transitions. It also needs to be fitted into the existing sentence so that the new word blends in seamlessly. Words are pronounced with different emphasis and intonation depending on where they fall in a sentence, so context is important.

Advanced VoCo editors can manually adjust pitch profile, amplitude and snippet duration. Novice users can choose from a predefined set of pitch profiles (bottom), or record their own voice as an exemplar to control pitch and timing (top). (credit: Professor Adam Finkelstein)

For clues about this context, VoCo looks to an audio track of the sentence that is automatically synthesized in artificial voice from the text transcript — one that sounds robotic to human ears. This recording is used as a point of reference in building the new word. VoCo then matches the pieces of sound from the real human voice recording to match the word in the synthesized track — a technique known as “voice conversion,” which inspired the project name, VoCo.

In case the synthesized word isn’t quite right, VoCo offers users several versions of the word to choose from. The system also provides an advanced editor to modify pitch and duration, allowing expert users to further polish the track.

To test how effective their system was a producing authentic sounding edits, the researchers asked people to listen to a set of audio tracks, some of which had been edited with VoCo and other that were completely natural. The fully automated versions were mistaken for real recordings more than 60 percent of the time.

The Princeton researchers are currently refining the VoCo algorithm to improve the system’s ability to integrate synthesized words more smoothly into audio tracks. They are also working to expand the system’s capabilities to create longer phrases or even entire sentences synthesized from a narrator’s voice.

Fake news videos?

Disney Research’s FaceDirector allows for editing recorded facial expressions and voice into a video (credit: Disney Research)

A key use for VoCo might be in intelligent personal assistants like Apple’s Siri, Google Assistant, Amazon’s Alexa, and Microsoft’s Cortana, or for using movie actors’ voices from old films in new ones, Finkelstein suggests.

But there are obvious concerns about fraud. It might even be possible to create a convincing fake video. Video clips with different facial expressions and lip movements (using Disney Research’s FaceDirector, for example) could be edited in and matched to associated fake words and other audio (such as background noise and talking), along with green screen to create fake backgrounds.

With billions of people now getting their news online and unfiltered, augmented-reality coming, and hacking way out of control, things may get even weirder. …

Zeyu Jin, a Princeton graduate student advised by Finkelstein, will present the work at the Association for Computing Machinery SIGGRAPH conference in July. The work at Princeton was funded by the Project X Fund, which provides seed funding to engineers for pursuing speculative projects. The Princeton researchers collaborated with scientists Gautham Mysore, Stephen DiVerdi, and Jingwan Lu at Adobe Research. Adobe has not announced availability of a commercial version of VoCo, or plans to integrate VoCo into Adobe Premiere Pro (or FaceDirector).

Abstract of VoCo: Text-based Insertion and Replacement in Audio Narration

Editing audio narration using conventional software typically involves many painstaking low-level manipulations. Some state of the art systems allow the editor to work in a text transcript of the narration, and perform select, cut, copy and paste operations directly in the transcript; these operations are then automatically applied to the waveform in a straightforward manner. However, an obvious gap in the text-based interface is the ability to type new words not appearing in the transcript, for example inserting a new word for emphasis or replacing a misspoken word. While high-quality voice synthesizers exist today, the challenge is to synthesize the new word in a voice that matches the rest of the narration. This paper presents a system that can synthesize a new word or short phrase such that it blends seamlessly in the context of the existing narration. Our approach is to use a text to speech synthesizer to say the word in a generic voice, and then use voice conversion to convert it into a voice that matches the narration. Offering a range of degrees of control to the editor, our interface supports fully automatic synthesis, selection among a candidate set of alternative pronunciations, fine control over edit placements and pitch profiles, and even guidance by the editors own voice. The paper presents studies showing that the output of our method is preferred over baseline methods and often indistinguishable from the original voice.