Does anyone have experience with approaches for selecting the number of sparse principal components to include in a regression model?

  • $\begingroup$ I don't have experience with that specifically, but I would assume that cross-validation would be one good approach (as always). $\endgroup$ – amoeba Dec 29 '14 at 22:42

While I don't have direct insights about your question, I ran across some research papers, which might be of your interest. That is, of course, if I understand correctly that you are talking about sparse PCA, principal component regression and related topics. In that case, here are the papers:

| cite | improve this answer | |
  • 1
    $\begingroup$ I did not know about all of these references. They are very good - thanks. $\endgroup$ – Frank Harrell Dec 30 '14 at 11:59
  • $\begingroup$ @FrankHarrell: You're very welcome! Glad I could help. $\endgroup$ – Aleksandr Blekh Dec 30 '14 at 13:04

The cross validation results were also used to determine the optimal number of dimensions for the LSI space. Too few dimensions did not take advantage of the predictive power of the data; while too many dimensions resulted in over-fitting. Fig. 4 shows the distribution of average errors for models with different numbers of LSI dimensions. The models with four dimensional LSI spaces produced both the fewest average number of errors and the fewest median number of errors, so the final model was built using a four dimensional LSI space.


I can post a copy if you aren't an ieee member.

This is from a paper I wrote in undergrad. I had a problem where I needed to decide how many dimensions (Latent Semantic Indexing is similar to PCA) to use in my logistic regression model. What I did was pick a metric (i.e. the error rate when using a flagging probability of .5) and looked at the distribution for this error rate for different models trained on different number of dimensions. I then picked the model with the lowest error rate. You could use other metrics like area under ROC curve.

You could also use something like stepwise regression to pick the number of dimensions for you. What type of regression are you preforming specifically?

What do you mean by sparse btw?

| cite | improve this answer | |
  • $\begingroup$ Sparse PC is for example L1 (lasso)-penalized PCA. In ordinary PCA we can usually enter terms in order of variation explained. With sparse PCA things are a bit more erratic so selection is perhaps more difficult. $\endgroup$ – Frank Harrell Mar 5 '14 at 23:29
  • $\begingroup$ The question was specifically about sparse principal components, and this answer (good as it is) does not address it at all, so -1. $\endgroup$ – amoeba Dec 29 '14 at 22:44
  • $\begingroup$ Stepwise regression that chooses components based on associations with $Y$ will result in overfitting unless special penalty functions are incorporated. $\endgroup$ – Frank Harrell Dec 30 '14 at 12:01
  • $\begingroup$ @FrankHarrell that can potentially happen but is less prone to happen if you use AIC instead of R-squared $\endgroup$ – Andrew Cassidy Dec 30 '14 at 15:53
  • $\begingroup$ @amoeba I'm confused... no I didn't address the "sparse" part of the principal comments, but you made the exact same suggestion to use cross validation in a comment? $\endgroup$ – Andrew Cassidy Dec 30 '14 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.