Yan LeCun's paper "Efficient Backprop" indicates that the average of each input variable over training set should be close to zero.

If the input variables are all categorical variables and encoded by one hot encoding. In this case the averages are certainly not zero. So do I need to normalize this kind of vectors?

  • $\begingroup$ You probably mean standardize, rather than normalize. I only skimmed that paper once, but my instinct would be that you would still use the same input pre-processing, i.e. standardize (and decorrelate?). I don't think LeCun's arguments should change just due to boolean/binary input. (Of course the standard advice still applies: try it both ways and see!) $\endgroup$ – GeoMatt22 Sep 8 '16 at 3:49
  • $\begingroup$ Is there an advantage of having the inputs as one hot, instead of single numeric value? While using one hot does not need normalization, it feels like it's adding correlated features. If you decide to encode the categories with numbers, then normalization will be beneficial. $\endgroup$ – Iliyan Bobev Sep 8 '16 at 7:00

A standard approach to all-categorical inputs is to train an embedding to represent the categories as real vectors. A standard example of this approach in the context of natural language processing is


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.