5
$\begingroup$

I've been using SVM to classify a data set without applying PCA. The classification rate was not bad, but I thought maybe applying PCA increases performance.

I have a training set (without labels) with size of 700x60 (i.e. 700 feature vectors each comprises of 60 different features). I applied PCA to that matrix in Matlab, getting a 60x60 matrix.:

coeff = pca(trainVector);

But I am confused about what to do next. How should I utilize this information?

I applied PCA to only training set and have 60x60 PCA matrix only. What will be my new training set after PCA and what should I do with the test set? Something like transformation matrix? I wasn't doing something like cross validation. As I know, SVM algorithm already applies cross validation. Previously I used a SVM tool to get a prediction model and then use this Model and the SVM tool to classify test set. I wanted to improve the accuracy by using PCA before classification.

Improve this question
$\endgroup$
2
5
$\begingroup$

The purpose of PCA is to reduce the dimensionality of your data, in part to reduce the number of parameters and therefore the variance of your predictor. I think you are applying PCA to the transpose of the matrix you want to apply it to, because your result should be $700 \times n$ for some $n$ that you choose. I am not familiar with the matlab implementation. That being said what you should be doing with PCA and SVM is,

  1. Normalize Data: Center and scale your data. The transformation, $C$, that centers and scales your data should be stored in some fashion.

  2. "Train" PCA: PCA learns a variance maximizing linear transformation onto a lower dimensional space. Learn this transformation, and then apply it to your training data. This transformation, $B$, should be stored in some fashion.

  3. "Train" SVM: As you were already doing train SVM on your reduced data. The predictor $A$ should be stored in some fashion.

Classifying new data point, $x$, is then as easy as $A \circ B \circ C(x)$.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ +1. It's not transpose, it's okay, but pca() returns eigenvectors of the covariance matrix and not PCA scores. Which is what is needed here, because test data should be transformed with the same coefficients. Here is the code until the SVM step: trainMean = mean(trainData); trainPCA = pca(trainData); numDim = 5; reducedTrainData = bsxfun(@minus, trainData, trainMean) * trainPCA(:,1:numDim)'; reducedTestData = bsxfun(@minus, testData, trainMean) * trainPCA(:, 1:numDim)'; $\endgroup$ – amoeba Oct 5 '15 at 20:36
  • $\begingroup$ I'm not so sure about this. It doesn't look like you're centering the data before training the PCA. gotcha $\endgroup$ – jlimahaverford Oct 5 '15 at 20:38
  • $\begingroup$ pca function centers the data automatically. $\endgroup$ – amoeba Oct 5 '15 at 20:39
  • 1
    $\begingroup$ @amoeba I guess the last transpose sign of the script reducedTrainData = bsxfun(@minus, trainData, trainMean) * trainPCA(:,1:numDim)'; should be deleted because the matrix dimensions won't agree. Implementing that (without the last transpose sign) produces the numDim columns of the PCA scores. Actually the exact same matrix can be also achieved by using [coeff,score,latent]=pca(trainData). I will share the results once I finish. $\endgroup$ – medemirhan Oct 5 '15 at 22:12
  • 1
    $\begingroup$ @amoeba I got it. That's why you give the expansion reducedTestData = bsxfun(@minus, testData, trainMean) * trainPCA(:, 1:numDim); instead of using MATLAB's default pca() function. Great! $\endgroup$ – medemirhan Oct 5 '15 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.