Get Ready for the Next Level of Gaming

We have seen how new interfaces use human face, voice or even movement to

trigger an action. Sensors-based on such interfaces are part of an emerging

field of computing called Natural User Interface (NUI). These new technology

forms not only recognize face, voice or gesture of humans, but are also adaptive

to the person's mood, social context and location. The aim of natural user

interfaces is not to make keyboard and mouse obsolete, but to expand the

dimensions of the user experience to technology. At the exclusive media day

hosted by Craig Mundie, Chief Research and Strategy Officer, Microsoft Corp. on

the eve of Microsoft Tech Fest 2010, research work on next generation of

interfaces was showcased, and for the first time the technology behind the

much-talked Project Natal was revealed in detail. Project Natal is code name for

the Microsoft XBox 360 add-on upgrade that incorporates face, voice, gesture and

object recognition technologies into one and presents users with a new approach

to interact with the XBox gaming console without the need of any controller.

Microsoft expects to ship this new technology by end of this year. Nintendo's

Wii with its motion-sensing handheld “Wiimotes” was able to get the players off

their couch and onto their feet but still, the user had to use a controller.

Microsoft's approach is to eliminate the controller, and power the user with the

ability to manipulate game characters via body movements or gestures. Natal is

an add-on device that can integrate with existing XBox consoles. The device

consists of RGB depth sensor, which is an infrared projector that couples  with

a monochrome CMOS sensor. This sensor allows the camera to see in 3D in any

ambient light. With the monochrome camera in the image the area which is white

is nearest to the device while the area that's black is at the far end, the

shades of grey represent the area in between. With these different shades of

grey the device can judge the proximity of the user. The system is intelligent

enough to track each player and even if these players overlap, the system

deciphers body gestures from a player in multi-player mode. This is achieved by

the system through real-time motion capturing and by recognizing the different

body poses and then rendering them onto the game space. While doing so, the

system recognizes each player in front of the device by making a wire-frame scan

of them, and it creates a sphere around each player. The system concentrates on

the spatial skeletal joints to trace the body parts movement of the player. And

if the wire-frames overlap, the system intelligence helps to discard the body

posture of the wire-frame representation of the player that isn't possible.

Therefore if two players overlap in front of the camera, the device through

their wire-frame posture can decipher the player which represents a game

character. This way even without the controller, Natal can recoginize and keep

track of a player who moves around in front of the sensor camera even in a

multi-player mode. The system can simultaneously track 16 players.

The console picks up your body movement through which you can control the game using your body.

The author was hosted by Microsoft Research in Bellevue.