Data whitening for improving regression My Data: $X_i= \{0.4;~7;~1,000;0;~0.8;~1;~0;~40;~0.7;~1;~0;~89,100\},~Y_i=345$  The training set size is $\approx35, 000$.  $Y$ is the dependent variable and the task is to estimate its value (regression).  


*

*I normalized the $X$ data by scaling them and then applying a sigmoid function. I did the same with $Y$. I tried using a neural network with 3 layers:  


*

*1st input(neurons=number of features),  

*2nd - hidden=20 neurons,  

*3rd - output=1 neuron;  

*each layer has sigmoid function),  

*learning rate=0.9,  

*momentum=0.5,  


I also tried using SVM(kernel="radial") and glm(family="gaussian") models, but nothing performed well.  

*I applied a whitening procedure (to remove correlations) to $X,Y$ together as one matrix (by chance). And as the result the StdErr was less than 1%.  


So, the question is: in my test data I have only $X$ and usually whitening is applied only to $X$ (inputs). Is it possible to apply results from previous whitening to a new dataset (without known $Y$)?  Why did I get so much improvement?
 A: *

*Obviously I have no idea about your dataset, but a learning rate of 0.9 seems exceptionally large for MLP.

*Whitening makes the data more Gaussian which MLP and GLM need (I have no experience on SVM). I have never tried whitening with X and Y together though, maybe you can try whitening on X and Y separately. Also, it seems you have both continuous and categorical variables, you might want to treat them differently.

*You normally won't be able to (and so shouldn't) fit another whitening procedure on your test data. Just record what you have done to in-sample X and in-sample Y, and transform the same on out-of-sample X, and the predicted values.
A: Whitening de-correlates the data linearly. That makes the optimization of neural nets substantially easier; while neural nets per se are invariant towards linear transformations of the input data, the optimization is not. If the feature $x_1$ and $x_2$ are uncorrelated (and thus "somewhat independent" :) the corresponding weight vectors $w_{1, \cdot}$ and $w_{2, \cdot}$ will make up different terms of the equation you are solving. 
The effect of whitening might be not so substantial if you use more sophisticated or finely tuned optimizers.
For details, I recommend "Efficient Backprop" by Yann LeCun. He writes it better than I could.
