Since its unveiling at 2009’s E3, Microsoft Kinect (Or Project Natal, as it was originally known) has caused a bit of a stir. The Xbox’s foray into the motion-controlled gaming war is a controller-less one, which has left many sceptical as to its accuracy, and those that have tried it bewildered as to just how it works.

Microsoft invited T3 for a UK exclusive, all-access tour round the labs at the company's HQ in Redmond, Seattle to see exactly how Kinect works, and the work that’s gone into it. Kinect is made up of three distinct subsystems that each marries hardware and software, all of which are detailed below:
 

How does Xbox Kinect work? Movement tracking

Kinect’s optical setup is what allows it to track your movements in real time. It’s ridiculously complicated and made up of tech that's been around for about 15 years, but allows for effects and functions that have only been available at huge expense up until very recently.

It’s made of two main parts: a projector and an IR VGA camera. The former bounces out a laser (don’t worry, Microsoft insists it’s safe) across the entire field of play, which the camera picks up to separate you from your sofa on what’s called a ‘depth field.’ It’s essentially all the pixels that Kinect gets back as IR noise measured in varying colour dependant on how close they are to the system. That way bodies appear a bright shade of red, green etc, and things further away appear grey.

The software takes this image and runs it through a host of filters so that Kinect can work out what’s a person and what’s not. The system follows a basic system of guidelines, such as ‘a body is from x-foot tall to x-foot tall’ and ‘a person has two arms and two legs’ to work out that your coffee table or dog aren’t extra players. It’s also taught to be able to pick you out if you’re wearing baggy clothes or have hair coming over your shoulders. When we saw this as the developers see it, it was impressively accurate at sussing out each body part (right shoulder, for example) from not much information.

Once that’s sorted, it converts body part identification into a skeleton with moving joints. Kinect is preloaded with 200 common poses, so that it can fill in the blanks if you make a move that obstructs the cameras view of your entire skeleton. The only downside we could see was that fingers aren't mapped individually on the skeleton, meaning that those dreams holding a pretend gun and pulling the trigger for Kinect FPS games are over.

The system does all this continuously at 30fps.

What about that promo trailer where Kinect signs in players just by looking at them? We saw that work in real life. The reality is that you’ll need to go through an ‘enrolment’ process in for that to happen. It’s a short one, but works by mixing your skeletal measurements with some basic facial recognition software. Microsoft says that if you drastically change your appearance you’ll need to reenrol.