How to assess regression performance when using PCA components as response variables?

Question

I have a regression problem for multi-dimensional response, say like 1000 dimension response, without using PCA to the response, my mean square error for regression is like ~80. Then, I apply PCA to these 1000 dimensions, and use 3 of them which represent 90% variation of the response, and perform the regression again. Now the mse is ~8. So how can I interpret this results? Can I claim that using 3 PCA components can get better regression since mse is 8 which is significant smaller than 80?

Answer 1

If I understand correctly, you're doing principal component regression of the test data onto the principal components of the training data. If this is the case, you seem to have encountered overfitting; the phenomenon of fitting the noise as well as the data. This is one of the problems PCA was conceived to remedy.