How can one go about recognizing a kind of motion using 3D depth data? I'm using a Kinect device, and I'm currently extracting Joints, and Depth data unto probably a buffer data of 15 frames. This is done at 30 frames per second.
The whole point of it is to try and recognize an arm motion, like a punches.
What I currently have in mind is have a multidimensional feature vector of 15 frames x {joints, depth data}, or possibly even calculate an optical flow for a sparse feature set using the iterative Lucas-Kanade method with pyramids to reduce it to 14 frames x{joints, depth data}
Then maybe apply a PCA, but I'm not sure what kind of classifier to use, and if I use a neural network how to build one to train it
Any advice?
 A: You'll save yourself time by surveying the literature (preprint), rather than inventing a new algorithm, which will likely be no better. Abstract:

Analysis of human behaviour through visual information has been a highly active research topic in the computer vision community. This was previously achieved via images from a conventional camera, however recently depth sensors have made a new type of data available. This survey starts by explaining the advantages of depth imagery, then describes the new sensors that are available to obtain it. In particular, the Microsoft Kinect has made high-resolution real-time depth cheaply available. The main published research on the use of depth imagery for analysing human activity is reviewed. Much of the existing work focuses on body part detection and pose estimation. A growing research area addresses the recognition of human actions. The publicly available datasets that include depth imagery are listed, as are the software libraries that can acquire it from a sensor. This survey concludes by summarising the current state of work on this topic, and pointing out promising future research directions. For both researchers and practitioners who are familiar with this topic and those who are new to this field, the review will aid in the selection, and development, of algorithms using depth data.

You may also find this code helpful.
