0
$\begingroup$

If I have a train set (train.arff, 10 attributes) I perform a PCA and I save my data with respect to the new transformed variables (say I choose the two first attributes, combination of the original ones, that collect most of the variance), and call this transformed trainset "trainset-afterPCA.arff". Now I train a model using this file (which only has 2 attributes), and save it.

If now I have a new dataset, constructed with the original 10 attributes, and I want to use the model I built before to classify this new data, how do I have to proceed?

If I just try to test on this new dataset, train and test aren't compatible, right? If I ran PCA on the test set, the resulting new attributes won't be the same as the ones obtained in the training set. What should I do?

Update:

Just to show (part) of the Weka output:

eigenvalue  proportion  cumulative
2.31715   0.28964     0.28964   0.512ent-0.472Threshold+0.422impRes-0.335pssm-mut+0.28 pssm-wt...
1.72263   0.21533     0.50497   0.593pssm-mut+0.501pssm-wt+0.41 Threshold+0.403hyd+0.161sub...
1.31987   0.16498     0.66996   0.698vdw+0.628sub+0.219Threshold-0.168hyd+0.154ent...
0.88362   0.11045     0.78041   0.53impRes-0.51pssm-wt-0.478ent+0.346hyd+0.33 subs-score...
0.8404    0.10505     0.88546   0.605hyd-0.552impRes-0.319pssm-wt-0.26pssm-mut+0.235ent...
0.56935   0.07117     0.95663   -0.656vdw+0.531sub-0.449hy+0.207Threshold-0.145pssm-mut...

Eigenvectors
V1       V2       V3     V4      V5      V6 
-0.4716  0.4104  0.219  -0.0231  0.215   0.2071 Threshold
-0.153  -0.1263  0.6977  0.0049 -0.1865 -0.6556 vdw
 0.2465  0.4028 -0.1679  0.346   0.6055 -0.4486 hyd
 0.2511  0.1609  0.6277  0.3299  0.1513  0.5306 sub
 0.2799  0.5007  0.0529 -0.5097 -0.3189  0.061  pssm-wt
-0.335   0.593  -0.1004  0.0423 -0.2602 -0.145  pssm-mut
 0.5118  0.0616  0.1544 -0.4783  0.2354 -0.1246 ent
 0.4217  0.1459 -0.0799  0.5297 -0.5521 -0.0645 impRes

So, based on your answer, if now I choose V1 and V2 to represent my data, I have to use V1 and V2 and calculate the new attributes for the test set, previous to upload the test set to the model..

 V1 --> new att 1 = 0.512ent-0.472Threshold+0.422impRes-0.335pssm-mut+0.28pssm-wt...
 V2 --> new att 2 = 0.593pssm-mut+0.501pssm-wt+0.41Threshold+0.403hyd+0.161sub...
Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
1
$\begingroup$

The principal components you calculated from your test set are the axes of a two dimensional space into which you projected your training data. You need to project the test set into this space (defined by the principal components of your training data) to calculate predictions in your model. I don't have weka handy so I can't poke around the specific PCA implementation there, but the basic idea is that you need project the test data into the same space you projected your training data into.

I'm guessing that the weka output includes a matrix of your eigenvectors. If we call this matrix $M$ and your training data $X_{train}$, you project your training data into this space via:

$X_{train}^*=X_{train} M[,1:2]$

Just do the same thing to project your test data onto your principal components:

$X_{test}^*=X_{test} M[,1:2]$

Where $M[,1:2]$ is the matrix produced when we strip off all but the first two columns of $M$

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you @David Marx. I update part of the output of Weka. I didn't know whether Weka had a way to "transform" a test set based on previous eigenvectors or if I had to do it by myself. I think it's second option. Thanks : ) $\endgroup$
    – PGreen
    Commented Jul 31, 2013 at 10:20
1
$\begingroup$

A general purpose trick that is often useful is to use ordinary regression to predict each principal component from all the constituent variables, and to save the regression coefficients for later use in applying to a test sample. But note that split-sample validation often fails unless you have at least 10,000 observations in both samples, because of randomness inherent in making the split. Also you used the term 'classify' which implies that you are using an ultra-low-precision accuracy score (proportion classified correctly).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks @Frank. To make sure I understand your last sentence: do you mean "ultra-low-precision accuracy" because, based on my example, using V1 and V2 only would collect ~50% of tha variance? I put that example just to explain myself based on some specific data, but not to actually do it in this case. Thanks for your comments. $\endgroup$
    – PGreen
    Commented Jul 31, 2013 at 12:40
  • $\begingroup$ What I meant was that if you use proportional 'classified' correctly as an accuracy measure, that measure uses only 1 bit of information from the predictions so it is the lowest information content measure you can have other than no information at all. This is reflected in low precision (wide confidence interval) for the accuracy score. Proper scoring rules are much preferred (e.g., logarithmic probability score (deviance) or quadratic score (Brier)). $\endgroup$
    – Frank Harrell
    Commented Jul 31, 2013 at 12:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.