1
$\begingroup$

I'm doing a data analysis on data with more than 100 dimensions.

After that different ML-Algorithms like NN are applied to it.

When I do a PCA in the first place to reduce dimensionality to somewhat like 3-10, I persistently get better results (as in less miss-predictions) than without it.

My thought was that PCA should just speed up NN, etc, but not make them better?

Is this improvement realistic or did I make a mistake with my PCA?

This is how I´m doing it concretely:

Data; % training input
Test_Data; % test input
pca_size = 3; % pca size

%Scaling and Centering of Data
Scaled = (Data - mean(Data))./std(Data);

coeff = pca(Scaled);

Data_Reduced = Data * coeff(:, 1:pca_size);
Test_Data_Reduced = Test_Data * coeff(:, 1:pca_size);
Improve this question
$\endgroup$
13
  • 2
    $\begingroup$ PCA manages to keep most (if your dataset is structured) of the information in your dataset but condenses it in far less variables. Because of the curse of dimensionality your original dataset is not suited for classification, and because PCA condense that into few variables you end up with better results. Seems good to me. $\endgroup$
    – Riff
    Commented Jan 25, 2017 at 9:05
  • 2
    $\begingroup$ Reducing the dimensionality of your input reduces the complexity of your estimated neural network. It's quite plausible that if high variance components obtained from PCA are mostly what's important for classification, the reduction in variance (overfitting) due to a simpler model may dominate the increase in bias. $\endgroup$
    – Matthew Gunn
    Commented Jan 25, 2017 at 9:20
  • $\begingroup$ Added sample code how I´m doing PCA. Thank you so far. $\endgroup$
    – SwingNoob
    Commented Jan 25, 2017 at 9:22
  • $\begingroup$ See stats.stackexchange.com/questions/141864 and stats.stackexchange.com/questions/142557. It's all covered there. $\endgroup$
    – amoeba
    Commented Jan 25, 2017 at 9:31
  • 1
    $\begingroup$ It indeed is matlab code. And Data - mean(Data) does subtract the column mean. Just tried that out. Also verified the dividing by std(Data). Works perfectly fine. Calculates standard derivation of each column and divides each colum element by the corresponding standard derivation of that column. $\endgroup$
    – SwingNoob
    Commented Jan 25, 2017 at 17:25

0

Reset to default