Newton, Einstein Go Digital

April 16, 2007 — "For every action there is an equal and opposite reaction," says Isaac Newton, throwing his long wavy hair over a shoulder and adjusting the collar of his 17th-century shirt.

A 13-year-old boy wrinkles his brow and steps with baggy pants onto an orange skateboard.

"Don't understand? Let's use your skateboard as an example," says Newton, pointing toward the device invented centuries after his death.

These are not actors but rather a curious modern-day boy interacting with a life-like, computer-generated version of the famous physicist who authored the three laws of motion — among other things.

So far, that kind of conversation is still fiction. But it's the ultimate goal of a new research project that plans to merge gaming technology with artificial intelligence to build an archive of virtual figures that behave and respond as naturally as real people.

"We are for the first time creating technology that is focused on archiving people rather than artifacts," said Jason Leigh, associate professor and director of the Electronic Visualization Laboratory at the University of Illinois at Chicago. "These avatars will be able to field questions, provide opinions and make recommendations just like a human."

Leigh is collaborating with artificial intelligence scientists Avelino Gonzalez and Ronald DeMara from the University of Central Florida in Orlando over three years to create an avatar for a senior program manager at the National Science Foundation who has a wealth of institutional knowledge that could be useful to future program managers at the foundation. (The name of the program manager has not yet been officially announced.)

And creating digital archives of people — influential politicians, Nobel Prize winners, poets, CEOs, etc. — whose knowledge could be vital to future generations is just one application that Leigh envisions.

There are other potential commercial applications as well. For example, digital docents could provide a tour of a museum or give students a political or historical lecture.

"We want to preserve more than a person's factual knowledge. We want to preserve their mannerisms and personality so you feel that you are talking with a real person," said Andrew Johnson, co-investigator on the project.

The first step toward creating the avatars is to build a motion capture studio equipped with digital cameras. The cameras will capture the body and facial movements of real people. Virtual characters can then be built using the motion capture data and special modeling software.

The UIC team will pay special attention to improving the lip motions of the animated characters they create, so that spoken words are closely synched with mouth movements. By the end of the third year, they hope to have facial recognition technology at a point where it can analyze a person's expression and provide an appropriate response.

But gaming technology and artificial intelligence may not be enough to achieve such natural results, said Michael Zyda, director of the University of Southern California's GamePipe Laboratory. Emotions need to be folded into the model too, he said.

"It's a separate thing from artificial intelligence. What can we do in modeling to give you the illusion that there is an emotional state?" said Zyda.

"We need sensing technology that allows us to sense real humans' emotional state, and then we can build models for the virtual state," he said.

After all, an empathetic Newton makes for a better teacher.