How does PCA improve the accuracy of a predictive model? I've seen in a kaggle challenge about digit recognition someone who used PCA before decision tree or other techniques.
I thought it was just for compressing data but he aimed to improve his score.
How can PCA improve score in this case ? Is it because there is less overfitting ?
 A: Disclaimer: I'm usually wrong at things.
Decision trees, by virtue of doing recursive splitting of your samples, with splits being based on a single variable, can only generate decision boundaries parallel to the axes of your co-ordinate system. So by rotating the data to directions of maximum variance/diagonalizing your covariance matrix as best you can, it might be easier to put decision boundaries between your class distributions
That being said, I'm not sure why you'd do PCA (without discarding some of your eigenvectors) before using a neural network model or whatever, because the rotation alone makes no difference - the network can approximate any function through the feature space.
A: Dadi Perlmutter once said: "What is the difference between theory and practice?  In theory they are the same while in practice they are different".  This is one of those cases.
Methods like Neural Networks often use gradient descent derived methods.  In theory if you had infinite number of iterations and retries, the algorithm is going to converge to the same result independent of coordinate system.  Neural Networks do not like the "curse of dimensionality" and so using PCA to reduce the dimension of the data can improve speed of convergence and quality of results.  The transformation of the data, by centering, rotating and scaling informed by PCA can improve the convergence time and the quality of results.
In theory the PCA makes no difference, but in practice it improves rate of training, simplifies the required neural structure to represent the data, and results in systems that better characterize the "intermediate structure" of the data instead of having to account for multiple scales - it is more accurate.
My guess is that there are analogous reasons that apply to random forests of gradient boosted trees or other similar creatures.  (Link)
A: An insight I gained from Jonathon Shlens' "A Tutorial on Principal Component Analysis":
Performing PCA is like choosing a camera angle, to gain the best possible view of the variance to be explained. 
So I'm joining user1843053. At proper angle, decision boundaries parallel to the axes of new, rotated coordinate system might make more sense than in original feature space, allowing for better performance of e.g. decision trees, even without discarding "non-principal" dimensions.
A: The PCA is a change of variables, using the correlations explained by orthogonal directions.
Removing directions with non-representative corresponding correlation is like removing noise. You will only keep significant datas.
By the way, thanks for the site.
