I have calculated the scores of all the columns in my dataframe, which has 312 columns and 650 rows, using PCA. I used the following code:

plt.plot(np.cumsum(all_pca.explained_variance_ratio_) * 100)  
plt.xlabel('Number of components')
plt.grid(which='both', linestyle='--', linewidth=0.5)
plt.xticks(np.arange(0, 330, step=25))
plt.yticks(np.arange(0, 110, step=10))
plt.ylabel('Explained variance (%)')  
plt.savefig('elbow_plot.png', dpi=1000)

The result is the following image:

enter image description here

My main goal is to use only important features for Random forest regression, Gradient boosting, OLS regression and LASSO. As you can see, 100 columns describe 95.2% of the variance in my dataframe.

I have 2 Questions:

  1. Can I use this threshold (100 Columns) for backward feature selection?
  2. Is it best practice to use Random forest for feature selection and train a random forest on the result? Or it would be better to use Backward/forward selection?

1 Answer 1


First: Doing principal components regression is mostly an alternative to lasso, random forest, and so on, not a preliminary step.

Second, lasso is a method of adjusting OLS (or other regressions such as logistic) - it's a way of building a model and adjusting the parameter estimates to account for the model building. Having a list that has both "lasso" and "OLS" seems odd.

Third, backward selection is a bad method, whether you start with principal components or raw variables. P values will be too low, standard errors too small, and parameter estimates biased away from 0. (Lasso is one attempt to correct these problems).

Fourth, PC does not allow you to use the "important features". Each principal component uses all the variables. Admittedly, some will be weighted very lightly in some PCs, but those may be weighted heavily in other PCs. If you think you have latent variables then factor analysis is a better tool than PCA. If you want to use the components for regression, then partial least squares might be better.

Fifth, 100 PCs is far too many to be sensibly interpretable and 100 regressors with 650 rows is too many and risks overfitting.

I would take a step back and ask yourself why you have 312 columns, what they all mean, whether some are redundant, and so on. You could use some tools to help with this, starting with a correlation matrix, but it should also rely on substantive expertise.

  • 3
    $\begingroup$ To add to Peter’s excellent answer, a large part of the value of PCA is its provision of an ordering of components so that you don’t need to do unstructured variable selection. As long as you select features in strictly increasing order of variance explained you limit model uncertainty. For example, you could solve for k such that the AIC of PC1-PCk is a minimum. $\endgroup$ Commented Feb 13 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.