Currently I'm trying to work out a project where I would like to recognize movements from videos using machine learning and python.

What I've done so far is extracting the x and y values of body joints of each frame in about 150 videos for each of the joints and put those in an Excel file, that now looks like this:

#Frame   NoseX    NoseY   NoseC   NeckX   NeckY  NeckC  .... 
  1      xvalue  yvalue  cvalue  xvalue  yvalue  cvalue ....   
  2      xvalue  yvalue  cvalue  xvalue  yvalue  cvalue ....  
  3      xvalue  yvalue  cvalue  xvalue  yvalue  cvalue ....  

This shows an example of the Excel file, showing just 2/25 joints and 3/250 frames (give or take). Each video of the specific movement is put in its own sheet, so in the end about 150 sheets with each 76x250 columns and rows (~2.8 million data points).

But all this data (videos) is currently just one type of movement that I was aiming for as training data (?), so that whenever I provide the model with a new set of different x and y values (other movement type) it should be able to classify it the same as the first set.... or not!

So in short: my output should be able to predict if the type of movement in the video is the same as that from those in the 150 videos, derived from the x, y (and c) values.

The problem I'm facing is that all the examples/tutorials I've encountered are so that each line is a set of data with a label/target that you can run a machine learning algorithm over, but not a whole sheet that defines a label (type of movement). So how do you create 1 label defines by the data in a sheet, instead of a line/row?

I'm hoping that this makes any sense and that someone here has any clue how to tackle such a problem, or that I need to check a different approach to this idea?

  • $\begingroup$ Do you have an automated way of getting these values from the input video? $\endgroup$
    – Jon Nordby
    May 23, 2019 at 11:58

1 Answer 1


You create a new sheet (or .CSV file) on format:


The identifiers in the video column would here be a string that is the name of the corresponding sheet with data. And in movement are your class labels.

An alternative is to have one file (.npy, .hdf5, .csv) for each video instead of a sheet, but the principle is exactly the same. In that case the identifier just refers to a filename in a directory instead of a sheet name.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.