I have an original data set with a number of features N equal to 135 and a number of rows equal to 32000. The last column of the data set ( column 136 ) can take either -1 or 1 depending on the class of the row. Each of the features can take a binary value of 0 or 1 ( characteristic present or not).

The original data set looks like this:

    V1    V1   V3   V4   V5  ............ V136
    0     1    0    0     1                 -1
    0     1    1    0     1                 -1
    1     0    1    1     0                  1
    .     .    .    .     .                  .
    .     .    .    .     .                  .
    .     .    .    .     .                  . ( row 32000)

I am required to use PCA to reduce the dimentionality of my data, and then apply SVM for classification. when I used PCA to cover 96% of the variance, I got a number of principal componenents equal to 54 ( which is good, since I reduced the number of features from 135 to 54), and then I computed the reduced data matrix which looks like this:

    V1                V2                V3                 V4  .................  V54    Class
    1.89463604908794 -1.15458215287551 -0.577464752091365 -0.560850946997309               -1
   -0.170737500644998 0.190545093723431 0.451249831953581 1.44451854658279                 -1

When I run SVM on the original data set, SVM finishes quickly, however, when I run it on the reduced data set, it takes forever. A part of the problem is that the original data set was binary, and thus it takes less memory when loaded. However, because the reduced data set is composed of real number, it takes more memory. For comparison purposes, I also saved the reduced data set to disque, and I compared the size of the two files in terms of KB:

    Original Data ( not reduced) : 8500 KB
    Reduced Data set             : 32200 KB

It looks like PCA reduced the number of features but it increased the total size in term of KB. Any ideas about how I can overcome this issue? Doesn't this throw away the original purpose of PCA ( reducing the dimentionality, and thus the size)?

  • $\begingroup$ What is the "original purpose" of PCA? I'm not sure there is one exactly. More importantly, what is your purpose in doing PCA? If it is just to save KB of memory & make the SVM run faster, then yes, you should skip it. $\endgroup$ Nov 24 '13 at 22:52
  • $\begingroup$ PCA reduces the number of features needed to describe your data. The size is a side effect. $\endgroup$ Nov 24 '13 at 23:24
  • $\begingroup$ Can you give me more information of your PCA part? which function you are using prcomp? and then how you cut off at 96% variance? A reproducible sample code maybe.. with anonymous variables names will be extremely helpful $\endgroup$
    – B.Mr.W.
    Nov 25 '13 at 5:26
  • $\begingroup$ Actually, I did not use prcomp. I implemented my own version of PCA following the exact steps of tha algorithm ( computing the covariance matrix, and then taking the first "m" eigenvectors that are needed to compute the final reduced data set) $\endgroup$
    – John
    Nov 26 '13 at 1:22

You're confusing the dimensionality of your data set with its size. They get swapped all the time informally, but the difference matters here:

  • Your data set has a dimensionality of 32000 cases x 135 features.
  • The size of your data set is 8500 KB

Note that the units aren't compatible.

PCA reduced the dimensionality of your data from 32000 x 135 -> 32000 x 54 (2.5x, not too shabby). PCA is pretty agnostic to how the data is stored, and unfortunately, switching from boolean values (a smart program can fit 8 to a byte) to doubles (8 bytes/double) more than counteracts whatever savings you get from projecting your data onto a lower dimensional subspace.

If reducing the memory footprint of your data is the only reason you care, I'd just skip it. (It's only 8 Mb anyway!). However, if you're interested in feature selection, you could do a factor analysis and manually look for variables that are either poorly correlated with the target variable OR are highly correlated with many other variables. These could then be eliminated from your original data set. This might give you a modest reduction in its dimensionality, while keeping it as a collection of binary features.


i think ,you are questioning purpose of PCA by considering single case i.e. when your data set is having binary values. Now assume that your data set is having real values instead of binary values then i m sure PCA will do its task perfectly i.e. reducing the dimensionality, and thus the size

In your case, if PCA is increasing size of data set then don't use it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.