Joystick Takes Cues by Voice

Oct. 19, 2007 -- "Ah," "ee," "aw" and "oo" are not just vowel sounds, but vocalizations that can now control a computer cursor. The Vocal Joystick detects sounds at 100 times a second and then turns them into movements on the screen.

The device could help people with disabilities surf the Internet, draw, play video games, control an electronic wheelchair and even operate a robotic arm.

"If you have full vocal capabilities, it's very natural and doesn't require anything special," said Jeff Bilmes, an associate professor of electrical engineering at the University of Washington in Seattle.

The system is essentially a software program that allows users to specify any sounds they want and then associate them with movements otherwise controlled by joystick knobs or buttons.

For example, "ah" could drive the cursor up; "oo" send it down; "ee" make it go left and "aw" make it go right.

A hard "k" sound could do the same thing as clicking a mouse button and a "ch" could simulate releasing the button. Increase the volume and the cursor moves faster.

The technology calls to mind Victor Borge, the musician and comedian famous for his "Phonetic Punctuation" routine, in which he recited a story, making onomatopoetic sounds for the commas, periods, question marks, etc.

And it's no surprise.

"I've been a fan of his since I was a kid. He was a definitely a big inspiration for coming up with this idea," said Bilmes.

But the Vocal Joystick is not just about short vowel or consonant sounds to move a cursor. When paired with speech recognition, the technology could give an impaired person full command of a computer.

A person could, for example, ah and oo their way to an Internet site and "k-ch" into the search box. Once there, they could speak the search criteria. They could also draw circles or squares in a drawing program.

Using the vowel and consonant sounds is more useful than saying "move up" or "move down," which is not very practical, said Alex Acero, research area manager at Microsoft Research in Redmond, Wash., an expert in speech recognition and natural language processing.