Augmented Reality Can Help Children With Autism Tap Into Their Imaginations

Playing pretend as a child—whether using a paintbrush as a wand or imagining a large cardboard box to be a castle—is more than just for fun. It is also an essential developmental activity that teaches children social and emotional skills and builds their self-esteem. However, most children with autism—a neurodevelopmental condition that affects the ability to communicate and interact with others—are less engaged in imaginative play. And this can have a profound impact on them into their adult lives.

That’s why Ph.D. candidate Zhen Bai designed an augmented reality (AR) system that she hopes will nudge such children toward more imaginative play. Her system lets children see themselves on a computer screen as they would in a mirror. She then gives the children simple physical objects—foam blocks, for example—that appear on the screen as a car, train, or airplane. The system’s computer-vision program detects where and how the child moves the block and mimics the activity in the image on the screen.

Bai is a student at the Graphics & Interaction Group at Cambridge’s Computer Laboratory. Her research paper, “Using Augmented Reality to Elicit Pretend Play for Children With Autism,” was published in December in IEEE Transactions on Visualization and Computer Graphics. It is available in the IEEE Xplore Digital Library. Her paper describes her observational experiment, which confirmed that the AR system could engage children with autism in imaginary activities. And from what she learned, Bai is developing the system to help children move to more imaginative play without the assistance of the technology.

The AR system relies on a Web camera with a wide field of view, a computer and a 60-centimeter monitor, and those simple foam blocks. The blocks are marked with black and white patterns resembling the Quick Response bar codes put on tags placed on merchandise and marketing materials, for example, for customers to scan with their smartphones and get more information about the product. The computer detects the marks on the blocks in each frame captured by the Web camera, and calculates their position and orientation. The program then superimposes a 3-D model of the toy. The children then see themselves playing with the car on the computer screen and not the foam block, augmenting the child’s real-life activity. Bai designed her system using the open-source Goblin XNA library.

To read the rest of this article, published on The Institute, please click here.