Using PCA to reduce dimensionality for a predictive machine learning regression model?

Question

I'm currently working with a dataset of 4000 molecules and their values for a given physical constant. I use the molecules to calculate ~1400 descriptors, then use the descriptors to predict the constants via ML algorithms from scikit-learn.

When using PCA, my understanding is the principal components are constructed from the whole dataset - I can then split the principal components into training and testing sets for use with a regression model (eg scikit-learn's GradientBoostingRegressor).

If I want to use the resulting model to predict targets for new molecules with no associated constant, can PCA still be applied? I can still calculate the original 1400 descriptors, but I'm not sure if the PCA steps can/should be completed without retraining the predictive model on the new output.

Answer 1

A key idea to remember about a test or validation set in machine learning is that you use it to mimic the real application of using your model to make predictions on data where you truly don’t know the answer. Do your split; then ignore the holdout data for the remainder of the training. In your case, this is exactly what Cagdas Ozgenc described: diagonalize the covariance matrix of just the training data, and then use those eigenvectors on the holdout data to create the out-of-sample principal components.

Once you’re happy with how you’ve tuned your model, then you would combine all data and train the model on everything, but at that point, you’ve decided that you think the model is sufficiently general and will not overfit so badly. This would be like Apple training Siri on 800 in-sample data points and then validating on 200 out-of-sample data points. Then they figure out what model they want. Then they train the model parameters on all 1000 data points, and that model is what makes it into iPhones that people buy.