My Data: $X_i= \{0.4;~7;~1,000;0;~0.8;~1;~0;~40;~0.7;~1;~0;~89,100\},~Y_i=345$ The training set size is $\approx35, 000$. $Y$ is the dependent variable and the task is to estimate its value (regression).

  1. I normalized the $X$ data by scaling them and then applying a sigmoid function. I did the same with $Y$. I tried using a neural network with 3 layers:

    • 1st input(neurons=number of features),
    • 2nd - hidden=20 neurons,
    • 3rd - output=1 neuron;
    • each layer has sigmoid function),
    • learning rate=0.9,
    • momentum=0.5,

    I also tried using SVM(kernel="radial") and glm(family="gaussian") models, but nothing performed well.

  2. I applied a whitening procedure (to remove correlations) to $X,Y$ together as one matrix (by chance). And as the result the StdErr was less than 1%.

So, the question is: in my test data I have only $X$ and usually whitening is applied only to $X$ (inputs). Is it possible to apply results from previous whitening to a new dataset (without known $Y$)? Why did I get so much improvement?

  • $\begingroup$ Do you mean you have a single observation? Judging from your profile you should be able to edit this question to make it more precise. This is will help you get better answers. $\endgroup$
    – user603
    Commented Mar 2, 2013 at 13:25
  • 1
    $\begingroup$ Sure, no. I have ~35 000 observations (train set) $\endgroup$
    – Ivri
    Commented Mar 2, 2013 at 13:39
  • 1
    $\begingroup$ Welcome to the site, @Ivri. I took the liberty of editing your post to make it clearer & easier to read, please make sure it still says what you want it to say. $\endgroup$ Commented Mar 2, 2013 at 15:58

2 Answers 2

  1. Obviously I have no idea about your dataset, but a learning rate of 0.9 seems exceptionally large for MLP.

  2. Whitening makes the data more Gaussian which MLP and GLM need (I have no experience on SVM). I have never tried whitening with X and Y together though, maybe you can try whitening on X and Y separately. Also, it seems you have both continuous and categorical variables, you might want to treat them differently.

  3. You normally won't be able to (and so shouldn't) fit another whitening procedure on your test data. Just record what you have done to in-sample X and in-sample Y, and transform the same on out-of-sample X, and the predicted values.

  • $\begingroup$ 1/ I tried different values of learning rate, but it seems that it's not so crucial ; 2/ That's what I'm asking: I tried it separately and nothing worked, I'm curious why it worked out previous time when they were together 3/ About the variable: most of them are continuous, but there are few discrete having values e.g. 0,..,5. I scaled all of them. $\endgroup$
    – Ivri
    Commented Mar 2, 2013 at 16:30
  • $\begingroup$ Whitening is a linear transform and leaves the data as Gaussian as it is. $\endgroup$
    – bayerj
    Commented Mar 2, 2013 at 20:13
  • $\begingroup$ @bayerj a linear combination of non gaussians will generally be more gaussian by central limit theorem. High dimensional independent uniforms look nearer gaussian if you rotate to the diagonals. $\endgroup$
    – Corvus
    Commented Mar 8, 2013 at 22:21
  • $\begingroup$ You are probably not 100% wrong, but it is a hard to justify intuition. The central limit theorem does not apply everywhere. Especially in typical neural net input data and especially if the data is not independent (otherwise, no whitening was needed), as required by the weak form of the CLT. Furthermore, MLPs do not have any assumptions about the input distribution. They just work better from an optimisation stand point. $\endgroup$
    – bayerj
    Commented Mar 11, 2013 at 12:42

Whitening de-correlates the data linearly. That makes the optimization of neural nets substantially easier; while neural nets per se are invariant towards linear transformations of the input data, the optimization is not. If the feature $x_1$ and $x_2$ are uncorrelated (and thus "somewhat independent" :) the corresponding weight vectors $w_{1, \cdot}$ and $w_{2, \cdot}$ will make up different terms of the equation you are solving.

The effect of whitening might be not so substantial if you use more sophisticated or finely tuned optimizers.

For details, I recommend "Efficient Backprop" by Yann LeCun. He writes it better than I could.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.