3
$\begingroup$

Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of the original dataset remains in the resulting dataset. A common goal is keeping variance between 90% and 99%.

Is it considered a good practice to try different values of the k parameter (size of the resulting dataset's dimension) and then check the results of the resulting models against some cross-validation dataset in the same way as we do to pick good values of other hyperparameters like regularization lambdas and thresholds?

$\endgroup$

2 Answers 2

3
$\begingroup$

Unless you have reason to believe on a prior number of components (e.g. a hypothesis saying the number of dimensions must me around $K$ or so) tuning it is perfectly valid. Indeed, there's recent related research, for example: Selecting the number of principal components (Choi et. al, 2015)

This has also been discussed in this question. This post has a nice approach in Python, and this R function has a simple implementation for a tuning routine.

$\endgroup$
0
$\begingroup$

It's not a hyperparameter. If you pick some value $k$ for the dimension of the result, it simply means that you would be ignoring the remaining components. Training the model multiple times for the different values of $k$ would give the same results for all the components $<k$. So this is not something to tune in the regular sense. What you would do is to fit PCA to the data and then see how much variability is explained for different values of $k$ for the fitted model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.