My Data: $X_i= \{0.4;~7;~1,000;0;~0.8;~1;~0;~40;~0.7;~1;~0;~89,100\},~Y_i=345$ The training set size is $\approx35, 000$. $Y$ is the dependent variable and the task is to estimate its value (regression).

  1. I normalized the $X$ data by scaling them and then applying a sigmoid function. I did the same with $Y$. I tried using a neural network with 3 layers:

    • 1st input(neurons=number of features),
    • 2nd - hidden=20 neurons,
    • 3rd - output=1 neuron;
    • each layer has sigmoid function),
    • learning rate=0.9,
    • momentum=0.5,

    I also tried using SVM(kernel="radial") and glm(family="gaussian") models, but nothing performed well.

  2. I applied a whitening procedure (to remove correlations) to $X,Y$ together as one matrix (by chance). And as the result the StdErr was less than 1%.

So, the question is: in my test data I have only $X$ and usually whitening is applied only to $X$ (inputs). Is it possible to apply results from previous whitening to a new dataset (without known $Y$)? Why did I get so much improvement?

  • $\begingroup$ Do you mean you have a single observation? Judging from your profile you should be able to edit this question to make it more precise. This is will help you get better answers. $\endgroup$ – user603 Mar 2 '13 at 13:25
  • 1
    $\begingroup$ Sure, no. I have ~35 000 observations (train set) $\endgroup$ – Ivri Mar 2 '13 at 13:39
  • 1
    $\begingroup$ Welcome to the site, @Ivri. I took the liberty of editing your post to make it clearer & easier to read, please make sure it still says what you want it to say. $\endgroup$ – gung - Reinstate Monica Mar 2 '13 at 15:58
  1. Obviously I have no idea about your dataset, but a learning rate of 0.9 seems exceptionally large for MLP.

  2. Whitening makes the data more Gaussian which MLP and GLM need (I have no experience on SVM). I have never tried whitening with X and Y together though, maybe you can try whitening on X and Y separately. Also, it seems you have both continuous and categorical variables, you might want to treat them differently.

  3. You normally won't be able to (and so shouldn't) fit another whitening procedure on your test data. Just record what you have done to in-sample X and in-sample Y, and transform the same on out-of-sample X, and the predicted values.

  • $\begingroup$ 1/ I tried different values of learning rate, but it seems that it's not so crucial ; 2/ That's what I'm asking: I tried it separately and nothing worked, I'm curious why it worked out previous time when they were together 3/ About the variable: most of them are continuous, but there are few discrete having values e.g. 0,..,5. I scaled all of them. $\endgroup$ – Ivri Mar 2 '13 at 16:30
  • $\begingroup$ Whitening is a linear transform and leaves the data as Gaussian as it is. $\endgroup$ – bayerj Mar 2 '13 at 20:13
  • $\begingroup$ @bayerj a linear combination of non gaussians will generally be more gaussian by central limit theorem. High dimensional independent uniforms look nearer gaussian if you rotate to the diagonals. $\endgroup$ – Corone Mar 8 '13 at 22:21
  • $\begingroup$ You are probably not 100% wrong, but it is a hard to justify intuition. The central limit theorem does not apply everywhere. Especially in typical neural net input data and especially if the data is not independent (otherwise, no whitening was needed), as required by the weak form of the CLT. Furthermore, MLPs do not have any assumptions about the input distribution. They just work better from an optimisation stand point. $\endgroup$ – bayerj Mar 11 '13 at 12:42

Whitening de-correlates the data linearly. That makes the optimization of neural nets substantially easier; while neural nets per se are invariant towards linear transformations of the input data, the optimization is not. If the feature $x_1$ and $x_2$ are uncorrelated (and thus "somewhat independent" :) the corresponding weight vectors $w_{1, \cdot}$ and $w_{2, \cdot}$ will make up different terms of the equation you are solving.

The effect of whitening might be not so substantial if you use more sophisticated or finely tuned optimizers.

For details, I recommend "Efficient Backprop" by Yann LeCun. He writes it better than I could.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.