I am working on a project where I want to classify what's currently on a frame.

For example, given a video of a TV recording, you would have classes as: Ads, Show Opening, Show.

I have multiple video files, around 20~30 minutes, at 24 FPS, a 20 min video generates 28800 frames among unbalanced classes (more data points from Show > Ads > Show Opening)

I started testing a LinearSVC in a single video which provided good results, but adding more videos makes my data set too large, and a can't fit a LinearSVC in batches (I'm using scikit learn)

I'm wondering which steps to take, mostly:

  • Discard data points to make my train data-set smaller? Should I discard randomly, or discard similar data points?
  • Should I explore other ML classifiers or move to Neural-nets? Are there specific techniques for video processing? Currently I'm treating the video classification as several image classification problems, should I model my problem differently?

Reducing the size the data per video could help you. Here are some options:

  1. Try subsampling your video - for example, selecting frames at 1fps instead of 24fps. You could develop a more sophisticated subsampling method if your class imbalances are problematic.
  2. 28800 data points I assume is the number of frames? If so, then each frame will actually contain ~(H, W, 3) data points. You could consider resizing them or center cropping, or you could consider extracting lower-dimensional secondary features (like color histograms).

By "several classification problems", do you mean that you're creating an independent classifier for each class? sklearn.svm.LinearSVC has a multi_class argument you could investigate if you want to learn a single classifier.

As for other ML classifiers, I can only speak as to neural networks. I'd suggest loading a pre-trained model (like InceptionV4, which tensorflow makes available), getting the final features, and training a classifier on those features (could be a logistic regression as in typical NNs, but could also be sklearn.svm.LinearSVC or similar).

  • $\begingroup$ By "several classification problems" I mean, I'm classifying each frame independent of the previous or next frame. I was wondering if there's are a video classification technique where the past frames could be taken into account, if so, how it works $\endgroup$ – RMalke Jul 26 '18 at 18:45
  • $\begingroup$ Ah I see. I'm mostly familiar with the DL lit, so I'd point you at 3D convolutional networks. (arxiv.org/pdf/1705.07750.pdf)[This one] is available as a (github.com/deepmind/kinetics-i3d/tree/master/data/checkpoints/… tensorflow checkpoint]. As for image models, you can just extract the features (which span multiple frames) and use those as classifier inputs. If you don't want to do DL, you might be able to get away with reducing frames to color histograms, then using short sequences of histograms as inputs to your classifier. $\endgroup$ – Gianni Jul 26 '18 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.