Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have got the following problem at hand.
I have variable length videos which belong to one of the four classes $A,B,C,D$.
From each frame of a video, I extract a feature vector of length $N$. Hence, for a video of $K$ frames, I've got $K$ such feature vectors.
I'm told that LSTM (Long Short Term Memory) is good for sequence classification. The output of the classifier should be one of $A,B,C,D$. How should I create this model in LSTM?
Input: $K$ vectors of size $N$
Output: $A | B | C | D$
There is some literature to suggest a protocol for this. One paper that is particularly interesting for a first in the analysis of 2-D video images is this one, Software Analysis of Mining Images for Objects Detection:
http://actamont.tuke.sk/pdf/2013/n1/8licev.pdf
Here's the abstract:
The contribution is dealing with the development of the new module of robust FOTOM system for editing images from a video or mining image from measurements for subsequent improvement of detection of required objects in the 2D image. The generated module allows create a final high-quality picture by combination of multiple images with the search objects. We can combine input data according to the parameters or based on reference frames. Correction of detected 2D objects is also part of this module. The solution is implemented into FOTOM system and finished work has been tested in appropriate frames, which were validated core functionality and usability. Tests confirmed the function of each part of the module, its accuracy and implications of integration.
One possible barrier to implementing this approach could be their use of the proprietary FOTOM system.
A more directly relevant approach uses recurrent neural networks, Action Classification in Soccer Videos with Long Short-Term Memory Recurrent Neural Networks:
http://liris.cnrs.fr/Documents/Liris-4742.pdf
Here's the abstract to this one:
In this paper, we propose a novel approach for action classifi- cation in soccer videos using a recurrent neural network scheme. Thereby, we extract from each video action at each timestep a set of features which describe both the visual content (by the mean of a BoW approach) and the dominant motion (with a key point based approach). A Long Short-Term Memory-based Recurrent Neural Network is then trained to classify each video sequence considering the temporal evolution of the features for each timestep. Experimental results on the MICC-Soccer-Actions-4 database show that the proposed approach outperforms classification methods of related works (with a classification rate of 77 %), and that the combination of the two features (BoW and dominant motion) leads to a classification rate of 92 %.
Both papers seem to lead to promising results.
