6
$\begingroup$

I have some doubts regarding PCA. I have 318 observations with 10 variables, and one response, which is the dataset I want to use for training and building the model. I did the PCA and found that the first 5 PCs account for 96% of the variance, so I plan to use these 5 for the model. In Matlab help the regression is done as follows: 

betaPCR = regress(y-mean(y), PCAScores(:,1:5));

But the coefficients are then transformed to the original uncentered variables:

betaPCR = PCALoadings(:,1:5)*betaPCR;
betaPCR = [mean(y) - mean(X)*betaPCR; betaPCR];
yfitPCR = [ones(n,1) X]*betaPCR;

My question is: How can I find the model and how can I apply it to new, test dataset (100 observations with 10 variables and one response)? Should I use the transformed coefficients, and would the predicted y be centered? How can I use the calculated betaPCR on new data?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
5
$\begingroup$

It seems that you have 418 cases, divided into a training set of 318 cases and a test set of 100 cases. I'll answer your question and suggest a closely related but potentially better approach to your problem.

As noted on the MATLAB help page, for PCR it's best if the predictors are both centered and scaled to unit variance so that differences in scales don't unduly weight the results. They didn't do that in their example, but if your predictors are on different scales then you might consider scaling in addition to centering.

The code you adapted from that page apparently returns the regression to the original scales of both $x$ and $y$, according to how the plot on that page was obtained (although my memory of MATLAB syntax is too rusty to verify that directly). Replacing the $X$ in the last line of code with your matrix of test data should give you the predicted $y$ values for those test data. To verify, try using a small sample of your original data for $X$ and see if the predictions make sense.

But PCR with a training set and a separate test set might not be the best approach to your problem. PCR picks the principal components that capture the most variance in the predictor variables, but not necessarily those most related to the response variable. The partial least squares approach illustrated on the same help page can be better related to outcomes. Also, the separation into training and test sets, if that's what you've done, doesn't efficiently use all the information in your data.

Ridge regression, provided by the ridge function in MATLAB, is essentially PCR but with different weights placed on the components instead of the all-or-none selection in PCR. Large regression coefficients that don't help much with predicting outcomes are penalized. This helps bring the coefficients better in line with relations to the outcome variable and helps correct for overfitting. You can start with all of your data to set up the model (if my understanding of what you've done is correct), then use cross validation or bootstrapping to choose the penalty that minimizes prediction error.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you for giving me new idea, to try to use ridge regression. $\endgroup$
    – sandra011
    Commented Sep 8, 2015 at 11:05
2
$\begingroup$

Here are some ways to test whether your understanding of the calculations is correct:

  • Take a few of the training cases and calculate the prediction as you think. Then compare with the fitted values from the help page.
  • If you use the full PCA model (all loadings), the PCA performs only a rotation of the data. The predictions based on all scores should match the predictions by least squares regression on the original data.
  • You can do the PCR in one or two steps: to find out whether your one-step prediction directly from original data to the fitted value is correct, compare it with the two step procedure of first calculating scores for the new data followed by predicting y from the scores.

Note that if the PCA fitting routine centers (and possibly scales) the data, you need to do that with your new cases as well. And yes, if the regression is set up to predict y - mean (y), then the predicted values will be centered by the mean of the ys. You then need to center and possibly scale X by the center and scaling calculated on the training X, the same applies to y.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you very much for the response.. After several tries, my current consclusion is that :before I do PCA I have to standardize and centralized X_train, then do regression on y_train-mean(y_train), and if I want to use those regression coeff to predict on new data (X_test), I need to use mean(X_train) and std(X_train) for standardizing X_test, calculate Z_test, use previous regression coefficients and add mean(y_train) to the result? $\endgroup$
    – sandra011
    Commented Sep 8, 2015 at 11:02
  • $\begingroup$ This sounds sensible. de.mathworks.com/help/stats/pca.html?searchHighlight=pca suggests that by default the data is mean centered (which you can switch off if you decide that another center is more appropriate for your data!) but not standardized. $\endgroup$
    – cbeleites
    Commented Sep 9, 2015 at 11:20
0
$\begingroup$

I am not a Matlab user, but you are approaching the problem from the wrong angle. PCA should not be used to help with overfitting - regularization is a proper tool for it. This way you are not throwing data away, and you will most likely end up with a model that is easier to explain (e.g. feature importance or weights, for your original variables, not transformed ones).

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for the response. Yes, but I still struggle with transforming new input dataset into principal components in order to check the model on test data. $\endgroup$
    – sandra011
    Commented Sep 6, 2015 at 23:07
  • 3
    $\begingroup$ -1: PCA is one possibility to regularize. $\endgroup$
    – cbeleites
    Commented Sep 7, 2015 at 11:16
  • $\begingroup$ this is off-topic, but why would you ever use PCA to avoid overfitting, when you can use proper regularisation techniques? The only benefit of pre-processing with PCA I can think of is to simplify/speed-up calculations down the chain. One can use PCA, but most likely should not, at least in my experience. $\endgroup$
    – volodymyr
    Commented Sep 7, 2015 at 13:31
  • 1
    $\begingroup$ Whether or not PCA is "the" or "a" proper regularization technique IMHO depends very much on the application/data generating process and also on the model to be applied after the PCA preprocessing. So without knowing more details, we cannot judge whether it is more or less appropriate than another regularization technique. For example, I regularly work with data (vibrational spectra) for which PCA regularization is usually much more appropriate than, say, LASSO regularization. Though e.g. for microarray data, it would be the other way round. $\endgroup$
    – cbeleites
    Commented Sep 9, 2015 at 11:07
  • $\begingroup$ Regularization will always "throw away information" in the sense that it introduces a bias which is accepted as "payment" for the reduction in variance. IMHO, the point is whether the particular regularization technique manages to throw away information that doesn't help solving the problem at hand without throwing away too much information that does help with the problem. $\endgroup$
    – cbeleites
    Commented Sep 9, 2015 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.