The size of my reduced data set is greater than the original I have an original data set with a number of features N equal to 135 and a number of rows equal to 32000. The last column of the data set ( column 136 ) can take either -1 or 1 depending on the class of the row. Each of the features can take a binary value of 0 or 1 ( characteristic present or not).
The original data set looks like this:
    V1    V1   V3   V4   V5  ............ V136
    0     1    0    0     1                 -1
    0     1    1    0     1                 -1
    1     0    1    1     0                  1
    .     .    .    .     .                  .
    .     .    .    .     .                  .
    .     .    .    .     .                  . ( row 32000)

I am required to use PCA to reduce the dimentionality of my data, and then apply SVM for classification. when I used PCA to cover 96% of the variance, I got a number of principal componenents equal to 54 ( which is good, since I reduced the number of features from 135 to 54), and then I computed the reduced data matrix which looks like this:
    V1                V2                V3                 V4  .................  V54    Class
    1.89463604908794 -1.15458215287551 -0.577464752091365 -0.560850946997309               -1
   -0.170737500644998 0.190545093723431 0.451249831953581 1.44451854658279                 -1
    ......
    ......

When I run SVM on the original data set, SVM finishes quickly, however, when I run it on the reduced data set, it takes forever. A part of the problem is that the original data set was binary, and thus it takes less memory when loaded. However, because the reduced data set is composed of real number, it takes more memory. For comparison purposes, I also saved the reduced data set to disque, and I compared the size of the two files in terms of KB:
    Original Data ( not reduced) : 8500 KB
    Reduced Data set             : 32200 KB

It looks like PCA reduced the number of features but it increased the total size in term of KB. Any ideas about how I can overcome this issue? Doesn't this throw away the original purpose of PCA ( reducing the dimentionality, and thus the size)?
 A: You're confusing the dimensionality of your data set with its size. They get swapped all the time informally, but the difference matters here:


*

*Your data set has a dimensionality of 32000 cases x 135 features.

*The size of your data set is 8500 KB


Note that the units aren't compatible.
PCA reduced the dimensionality of your data from 32000 x 135 -> 32000 x 54 (2.5x, not too shabby). PCA is pretty agnostic to how the data is stored, and unfortunately, switching from boolean values (a smart program can fit 8 to a byte) to doubles (8 bytes/double) more than counteracts whatever savings you get from projecting your data onto a lower dimensional subspace.  
If reducing the memory footprint of your data is the only reason you care, I'd just skip it. (It's only 8 Mb anyway!). However, if you're interested in feature selection, you could do a factor analysis and manually look for variables that are either poorly correlated with the target variable OR are highly correlated with many other variables. These could then be eliminated from your original data set. This might give you a modest reduction in its dimensionality, while keeping it as a collection of binary features.
A: i think ,you are questioning purpose of PCA by considering single case i.e. when your data set is having binary values.
Now assume that your data set is having real values instead of binary values then i m sure PCA will do its task perfectly i.e. reducing the dimensionality, and thus the size
In your case, if PCA is increasing size of data set then don't use it. 
