1
$\begingroup$

I am working on neural networks and I am currently creating a perceptron that will work as a classifier for a data set of images with faces. I am required to perform pca (principal component analysis) to my data set before dividing the samples into two different sets for training and testing. By doing this, I am lowering the dimensionality of the data and at the same time I am compressing the size of the images.

However, I am not a statistician and I have some problems defining the number of principal components to use for the pca method without any specific formula. My data set is an array of 4096x400, 400 being the number of the sample images and 4096 being their dimension. Is there a way to be more precise and accurate about the number of principal components to use during pca?

I am working on matlab so I am using princomp.

$\endgroup$
1
$\begingroup$

The standard procedure is to calculate the PCA decomposition, and to keep those components which account for, say, 95% of the total variance. In other words, you calculate the eigenvalues and order them decreasingly. Keep those components for which the sum of the corresponding eigenvalues accounts for 95% of the total.

If you would like to go for a more principled approach, you may consider probabilistic PCA, which has some additional advantages: you can handle missing values and is more robust when working with high dimensional data and relatively small data sets.

An extension of this idea (Bayesian PCA) allows you to select the number of components from the data automatically.

$\endgroup$
4
$\begingroup$

If you are using a single layer perceptron model, then I would advise against PCA and just add a regularisation term (weight-decay) to prevent over-fitting the training data. Regularised linear models generally don't have problems dealing with high dimensional data, provided the regularisation parameter is set carefully (e.g. my minimising the cross-validation error), and if you are unlucky, the discriminant information might lie in the low variance components that you discard using PCA.

My advice would be to set the number of components by minimising the cross-validation error, rather than rules-of-thumb, which may work well on average, but very badly on your particular problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.