The first big hit in gesture control technology was the mouse. If you’re too young to have any pre-digital memories, this might seem like an odd claim. My three-year-old is no more mystified by mice and touchpads than she is by building blocks. Once upon a time, though, we needed lessons in how the motion of a peripheral device rolling around on the table related to the motion of something on the screen called the “pointer.”
We caught on quickly, of course. But it was alien territory for a while. I bring this up because in some circles these days, gestural control of digital devices is being spoken of as the next big thing. At the same time, others have wondered about the feasibility of the concept. Will I accidentally erase my Word document by scratching my nose? Will I have to learn some weird new language of gestures? Is anyone really going to bother? (The image above is from a design Nokia patented in 2008, illustrating one of many proposed gesture vocabularies.)
Prognostication is a mug’s game, but it’s not too hard to imagine that what happened with the mouse could happen with today’s emerging gesture control technologies. The success of the Wii, which just officially became the best-selling Nintendo home console of all time, augurs well. Several companies are angling for dominance in a still very young market of Wii competitors, gesture applications for mobile, gesture-based home entertainment systems, and the like.
VentureBeat has covered many of these companies. The buzz around new products at the recent Consumer Electronics Show and the excitement about the upcoming release of Microsoft’s Project Natal — a gesture, voice and face recognition system for the Xbox 360 game console — suggest that this is the year when gesture control may finally arrive. So here’s a brief tour of what’s out there, what’s coming, and who seems likely to lead the way into a future when point and click may take on a whole new literal meaning.
I’ll profile a few developers first, followed by some commentary on the platform players.
Developers
Canesta
This Sunnyvale, Calif. company has been around since 1999. It makes low-cost CMOS 3-D imaging chips that enable webcam-like cameras to sense motion. Canesta has raised an impressive $74 million in financing and holds 40 patents. The company has been working with GestureTek and Hitachi to develop gesture controlled televisions and other devices. On Feb. 3, Canesta announced that Ed Fries, the former head of Microsoft Game Studios, would be joining its board of directors. The company has garnered excitement with its trade show demonstrations and seems likely to be a major player in the next couple of years, when its technology finally makes its way into consumer devices. Here’s a demo of Canesta’s television interface.
GestureTek
By far one of the earliest entrants in the field, Silicon Valley-based GestureTek was founded in 1986. The company holds patents on gesture and motion control technology for use in phones, games, PCs, toys, and other devices. GestureTek’s 3-D, multitouch, and gesture-controlled interactive displays are used in a wide variety of settings, including television weather reports, digital signage, tabletop consoles, and large interactive museum exhibits. Both Xbox 360’s webcam accessory and Sony’s EyeToy make use of GestureTek technology. GestureTek’s Eyemo software turns your phone into a kind of Wii controller. Users perform various motions such as shaking, rocking, and rolling in order to answer a call, play games, scroll through menus, or interact with browsers and maps. This video demonstrates GestureTek’s new Momo SDK, which debuted at CES 2010.
Softkinetic-Optrima
These two Belgian companies, both focused on device-free gesture technology, entered into a joint venture in November 2009 to market 3-D motion control hardware and software. The venture is co-managed by Optrima CEO André Miodezky and Softkinetic CEO Michel Tombroff. Optrima, founded in April 2009, makes 3-D CMOS sensors and 3-D cameras, and Softkinetic, founded in 2007, develops 3-D gesture recognition middleware and software tools for 3-D cameras. Softkinetic-Optrima has started up a new division called Softkinetic Studios that develops games and other applications for motion-sensitive 3-D cameras. Tombroff told me that the two companies see video games and television as the main areas where gesture control will remain focused and begin to gain traction. On the television front, Softkinetic-Optrima has just announced a partnership with the huge European-based engineering consultancy company Devoteam to “create a prototype for the next generation TV interface.” Here’s an interesting video showing physical rehabilitation games for the elderly that Softkinetic created in partnership with Silverfit.
Hillcrest Labs
The company’s cool Loop pointer, a circular wireless control device, went to market in mid-2009 amid positive reviews. Conceptually similar to a Wii remote, the Loop improves on Nintendo’s technology by cancelling out your hand’s natural tendency to shake, enabling more precise pointer movements onscreen. The Loop is sold for $99 at Amazon. Loop-like devices will likely compete with device-free gesture control for market share in the home media space. Hillcrest has licensed its technology to remote control maker Universal Electronics. Last year Hillcrest sued Nintendo, claiming the Wii remote infringed on four of its patents. Some have speculated that Nintendo signed a license deal with the company as part of the resulting settlement. Here’s GeekBrief’s video review of the Loop.
Kakai
Not much is known yet about this stealthy Santa Clara startup, though it has 40 employees and has raised over $9 million in funding over the last several months, giving rise to much curiosity and speculation. However, some clues such as a recent job posting suggest that it may be working on gesture control applications for consumer devices.
Mgestyk
Mgestyk combines a 3-D camera with “hand gesture language processing” to enable gesture control of Windows applications. The company claims its technology can perform well in low light conditions, is responsive to subtle gestures, and can be set up to control any existing Windows application without modification. Users learn a special language of gestures developed by Mgestyk. Mgestyk’s technology can be seen in action here.
PrimeSense
Based in Tel Aviv, 3-D camera maker PrimeSense is one of the vendors behind Microsoft’s Project Natal, the device-free gesture control system for Xbox 360 that is set to debut this year. Microsoft purchased 3DV Systems, another Israeli 3-D camera maker, back in February 2009, and then licensed technology from PrimeSense (our own sources tell us this, though Microsoft has not disclosed it) and GestureTek. The result is supposed to be fine-tuned gesture control coupled with face and voice recognition, a package that, if its promise becomes a reality, would go way beyond the Wii’s capabilities.
Sixense
Sixense, based in Los Gatos, Calif. and headed by Amir Rubin, has developed a tracking technology it calls TrueMotion, which enables gesture-controlled gaming using Wii-like wireless controllers. In contrast to inertial systems that rely on sensing the user’s motion and direction, Sixense claims its magnet-based technology (no cameras required) can pinpoint the user’s position in space and provide fine-tuned motion control down to the millimeter. The technology works with existing games, and Sixense recently announced a partnership with gaming hardware maker Razer to market a motion sensitive controller for the PC. Razer and Sixense have an SDK so developers can create new games with the technology. Sixense is a self-funded company with an A round “planned soon,” Rubin told me. This video from Chilla Frilla shows the Razer Sixense controller in action at CES 2010.
Omek InterActive
Like PrimeSense and 3DV Systems, Omek is based in Israel. The company makes software and develops games for use with 3-D cameras. Omek, which raised $3.3 million in 2009, offers an SDK called Shadow in addition to developing its own games. One interesting point about Omek is that its technology is not dependent on any particular 3-D camera, but will work with any depth-sensing camera on the market.
Ball-it
Privately funded Finnish company Ball-it is behind the Blobo game console, a small device shaped like a golf ball that went on the market in Europe in late 2009 and will soon launch in the U.S., according to an email from company CTO Johannes Väänänen. The Blobo console, in development since 2005, is a nifty little device that senses motion and proximity and can be thrown and squeezed for various kinds of game play. It’ll even count the calories you burn while playing games. VentureBeat’s Matt Marshall covered Ball-it back in 2008, and the core offering doesn’t appear to have changed much since then, but the company now has a market launch on its hands so it will be worth watching what develops. You can see a demonstration of the Blobo console here.
Platforms
Samsung
The electronics manufacturer is sitting on a patent for visual gesture control using the on-board camera in a cell phone along with specialized finger and hand motions. The patent diagrams show yet another specialized language of gestures. Samsung has done some work on gesture-controlled television, and back in July it made waves by exhibiting “gesture-sensing hologram” technology during the product launch of its new Jet phone.
Nokia
Aside from the 2008 patent mentioned above, the cell phone maker recently announced a new design for gesture control that employs radar in the phone. This technology is supposed to work even when the phone is covered by fabric — for instance, when it’s in your pocket.
Sony
Released back in 2003 for Playstation 2, Sony’s EyeToy is a specialized 2-D web camera made by Namtai and Logitech that provides game control using motion, sound, and color. Sales of the device have been fairly strong, but the technology is hampered by the fact that it isn’t very precise and doesn’t perform well in low-light conditions. Sony’s answer to these problems is a Wii-like motion control wand for Playstation 3 that was supposed to debut in the spring but was recently pushed back to fall 2010. A prototype looking like a “stunted light saber” was on display last June at the E3 show in Los Angeles. Here’s a video of that demonstration. The device will reportedly be named the PlayStation Arc.
Microsoft
Project Natal debuted at E3 last year as well, and has been the subject of near-incessant speculation before and since. Aside from promoting Natal for gaming, Microsoft would like to leverage its new gesture control system, now reportedly scheduled for release in time for Christmas 2010, as a way to make Xbox 360 the center of your home media universe. The idea is that you’d use gestures to control all of your media devices without the need for remote control. In addition, Natal is supposed to be able to do facial and voice recognition, meaning family members could have their own customized settings and features. The Natal hardware device is a narrow metal strip that rests near your television. The promo video, released last June, is quite cool and certainly worth a thousand words. There are some reports of gaming developers at work on Natal games, including recent news from Capcom.
Nintendo
Brandon Sheffield, editor in chief of Game Developer, opined in a recent editorial that Nintendo’s success with gesture control has more to do with marketing than with groundbreaking technology, and that Microsoft and Sony will be hard pressed to compete on those grounds. At the same time, cool technology counts for something, and some of the developers listed above seem to have a leg up on Wii when it comes to sensitivity and precision. Several companies, most notably Microsoft, are banking on the appeal of dropping the controller from the gaming experience entirely. In response to the sensitivity question, Wii released the MotionPlus add-on last year, but few games support it. In fact, development activity for Wii games appears to be stagnating. Wii may be the best-selling console in Nintendo history, but it remains to be seen how the console will perform in a market where it’s no longer the only game in town.
Apple
Apple recently secured a patent on proximity sensor technology, presumably for use with the iPad. The technology senses when a stylus, finger, or other object is near the display.
Finally, a couple of notes from the world of academia, where gesture control developments are occurring at something of a rapid pace. Back in December, I reported on an MIT research team’s debut of the BiDi screen, which tracks motion using a new type of LCD screen with embedded light sensors. Here’s the BiDi screen demo. In early January, researchers at the University of Tokyo’s Ishikawa Komuro Laboratory announced a new “in-air gesture input interface” that employs a special high-speed camera. You can see the prototype demonstrated here.
Both of these research projects, along with related developments like Nokia’s radar system, are targeted for use in cell phones, which would tend to suggest that the new wave of gesture control might soon converge with the rising tide of innovative mobile devices.