For extracting features from video frames (2 sample/sec) I use keras framework in python and load VGG16 that input size is (150,150,3) and output size is (4,4,512). After the feature extraction step I want to cluster frame features with Hierarchical K-Means.
My problems are as follow:
- I save each frame features in a vector which size is 8192. For a video that have 8000 frames if only reduce each frame size to (150,150) and extract features then I have a feature matrix with size (640,8192). As you can see feature matrix for even one video is very large ans besides "sparse". What is the best way to reduction its dimension?
- What is the best metric for calculation distance between two pair of frame features? The space is so sparse and even feature values are so small, so Euclidean Distance is not a wise choise!!
What is the frame feature:
As you already knew, videos are nothing but frames, and with the help of deep learning (VGG16 (without the last fully connected layer)) we can extract its features in the way we like. for more information kaggle.com/keras/vgg16
In this particular case, output features have the size of (4*4*512) that become 8192 number in a row vector.
My data as I mentioned above is a very sparse and large matrix (640,8192). Non-zero values are rarely up to 100.
For Dimension Reduction:
Two method are available for DR
- Principal component analysis (PCA): A statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. (source: https://en.wikipedia.org/wiki/Principal_component_analysis)
- Singular-Value Decomposition (SVD): A factorization of a real or complex matrix. It is the generalization of the eigen decomposition of a positive semi definite normal matrix (for example, a symmetric matrix with positive eigenvalues) to any m*n matrix via an extension of the polar decomposition. (source: https://en.wikipedia.org/wiki/Singular-value_decomposition)
Most important parameter of these two methods is "n_components" that is number of components to keep. This parameter have the value of min(n_samples, n_features). As you can guess, components that we kept with this module is depend of Sample Number and Feature Number. Suppose that we have two videos with feature matrices with size of (140, 8192) and (640, 8192). First element is number of frames and second element is number of features. The output of PCA for these two videos is (140, 140) & (640, 640). We have to have matrices with same axis to check distance and clustering. How to solve this problem?
I know that this clarification is too long to read, but it's worth it!