I have a task of classifying videos based on feature vectors generated for each frame in the videos. The individual features are numbers between 0 and 5, and the feature vectors have about 20 elements. There aren't that many videos to train on (and they vary in length) so I don't think just concatenating the framewise features would work. Currently (following the example of a paper on the subject) I'm generating a vector for each video that measures whether each feature exceeds a threshold in each frame of the video, but I expect there are other possible approaches (such as ones that don't throw away all the chronological information from the framewise vectors). What other kinds of things could I do? If it's relevant, my classifier is an SVM.