I saw this example in a python notebook on Fast.ai. In the notebook they are removing the background and keeping objects in the foreground in a video sequence by using different methods(SVD, random SVD, robust PCA etc). I tried to implement it using the regular PCA by doing the following:

  1. Reshape all frames in the video to vectors and concatenate the vectors into a matrix $A$.
  2. Calculating the covariance matrix of $A$ with dimension(no of frames x no of frames)
  3. Calculating the eigenvectors and eigenvalues by SVD (I'm only using the first principal component)
  4. Obtain the projection $Z =AU$
  5. Reconstruct the matrix with the background vectors. $\hat{X} = ZU^T$
  6. To get the foreground objects it is $A-\hat{X}$.

It works for me, although the result is not as good as robust-PCA. But i am not sure about what I am exactly are doing. I have read some a articles of post about PCA but I could not find some (that i understand) about background removal and PCA. So my questions are:

  1. Why is the PC1 the background of all the picture frames? And what is the intuition by calculating the eigenvectors of the covariance matrix?
  2. What happens if they camera shakes a little, so the positions of the background pixels change, can PCA still be used?
  3. Does it help to subtract the mean when working with images? (I could see that you are doing that in other PCA examples).
  • $\begingroup$ Note that the segmentation is about breaking consumers of a product or service into sub-groups. We don't yet have a specific tag for image segmentation, so image-processing will have to serve. $\endgroup$ – Sycorax Jun 16 '18 at 22:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.