How Disney Research is using augmented reality to give a 3D experience for coloring books

A coloring book and a box of crayons may give kids an early opportunity for creative expression but, next to TV and video games, coloring can sometimes seem unexciting. A coloring book app devised by Disney Research, however, can cause characters to leap from the page in 3-D glory with the help of augmented reality.

A child colors a character, such as an elephant, on the book page normally, while a tablet or smartphone running the app monitors the drawing. Based on the child's coloring, the app fills in colors in real-time on an animated 3D version of the elephant that is visible on the device's screen and integrated into the video.

The app keeps the core focus on the traditional activity of coloring while offering a magical digital overlay that enhances engagement.

In user testing - performed with adults rather than children in this early study - the researchers found that most users said the app increased their motivation to draw in coloring books and 80 percent said the app increased their feeling of connection to a character.

Researchers from Disney, ETH Zurich and the Swiss university EPFL presented the augmented reality app at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2015) in Fukuoka, Japan. Although the research work is just now being presented to scientific audiences, it has already gone through the tech transfer process, inspiring the commercial product called "Disney Color and Play" launched earlier this year by Disney Publishing Worldwide and Bendon.

This work fits into a larger initiative at Disney Research called Augmented Creativity, which focuses on using augmented reality to enhance creative play.

"Augmented reality holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity," said Robert W. Sumner, a principal research scientist who leads the group on animation and interactive graphics at Disney Research. "We are thrilled to have the opportunity to present the scientific advances behind this technology," continued Sumner, "and are especially happy that it is available to consumers, thanks to our cooperation with Disney Publishing."

To create this new experience, the researchers first created animated 3-D virtual characters and then use custom software to generate 2-D line-art representations of the characters for a coloring book. The app, operating on a device with a camera viewing the user and the coloring book, automatically detects the character the user is coloring and displays the 3-D version.

As the child applies color to the 2-D drawing, the app applies the same color to the 3-D character - both to the areas visible in the 2-D drawing and to the remainder of the 3-D form not visible in the book. Because the coloring occurs in real-time, the illusion is created that the user is also coloring the occluded areas, with similar texturing of the color.

Determining how to apply color to the occluded areas was one of the tougher problems to solve, Sumner said. Simply mirroring the user's colored strokes doesn't work because the pattern of colors used for, say, a character's face will not be appropriate for the back of the character's head. The color also has to be continuous, so no seams can be seen between the visible areas and the occluded areas or where disparate portions of the textures meet.

Their approach was to create a "lookup map" for each character, which matches pixels in the occluded parts with corresponding pixels in the portion visible to the user. User testing showed that this method provides better results than "naïve" approaches such as mirroring. Most importantly, the lookup map method enables the coloring to be performed instantly.

Because paper books don't lie perfectly flat and can flex as a user colors, the team developed a deformable surface tracking method to monitor surface changes and to keep the virtual character displayed on the device screen correctly oriented with the book's page.