What's a good approach of classifying video frames? I am working on a project where I want to classify what's currently on a frame.
For example, given a video of a TV recording, you would have classes as: Ads, Show Opening, Show.
I have multiple video files, around 20~30 minutes, at 24 FPS, a 20 min video generates 28800 frames among unbalanced classes (more data points from Show > Ads > Show Opening)
I started testing a LinearSVC in a single video which provided good results, but adding more videos makes my data set too large, and a can't fit a LinearSVC in batches (I'm using scikit learn)
I'm wondering which steps to take, mostly:


*

*Discard data points to make my train data-set smaller? Should I discard randomly, or discard similar data points?

*Should I explore other ML classifiers or move to Neural-nets?
Are there specific techniques for video processing? Currently I'm treating the video classification as several image classification problems, should I model my problem differently?

 A: Reducing the size the data per video could help you. Here are some options:


*

*Try subsampling your video - for example, selecting frames at 1fps instead of 24fps. You could develop a more sophisticated subsampling method if your class imbalances are problematic.

*28800 data points I assume is the number of frames? If so, then each frame will actually contain ~(H, W, 3) data points. You could consider resizing them or center cropping, or you could consider extracting lower-dimensional secondary features (like color histograms).


By "several classification problems", do you mean that you're creating an independent classifier for each class? sklearn.svm.LinearSVC has a multi_class argument you could investigate if you want to learn a  single classifier.
As for other ML classifiers, I can only speak as to neural networks. I'd suggest loading a pre-trained model (like InceptionV4, which tensorflow makes available), getting the final features, and training a classifier on those features (could be a logistic regression as in typical NNs, but could also be sklearn.svm.LinearSVC or similar). 
