I have had some success training my deep neural network (with ReLU hidden units) by first normalizing the features of my data set to zero-mean-unit-variance.

Each sample of my data set has 600+ values. Majority of them are continuous but a handful or columns are symbolic / discrete (represent by fixed range integers). Currently I just normalize all columns to zero-mean-unit-variance the same way. Should I only normalize the continuous value columns? Are there better data normalization strategies?

  • $\begingroup$ By continuous features do you mean something like spatial or temporal continuity? What type of data you are using? $\endgroup$ – yasin.yazici Jun 15 '15 at 13:06
  • $\begingroup$ They are financial models where numbers are percentage changes. $\endgroup$ – teddy Jun 15 '15 at 14:56
  • $\begingroup$ If adjacent features are mostly correlated (or continuous), I think that is also your case, I don't recommend feature based normalization. Instead sample based normalization would be better. That is simply subracting mean of each sample from itself and also dividing of each sample's std to itself. I'm not familiar with data type you mention, If you think that will eliminate some important information from the data, don't apply any normalization on continuous part. $\endgroup$ – yasin.yazici Jun 15 '15 at 15:35
  • $\begingroup$ thanks. will try your suggestion of sample based normalization. not normalizing seems to leave poor results though. $\endgroup$ – teddy Jun 16 '15 at 17:45
  • $\begingroup$ What do you mean by 'symbolic'? Are they categorical? If so, they should be converted to multiple binary variables. $\endgroup$ – pir Jun 17 '15 at 7:21

You said your integer-valued variables have a fixed range, which makes things relatively easy. You will need to encode them as several features rather than one.

  • If they are categorical (there is no inherent natural order within them), you can use one-hot encoding (i.e. the "fifth option" is represented as (0,0,0,0,1,0,0)), this is the common practice.
  • If they are ordinal (discrete, but with an order), using one-hot is possible, but it's even better to use (1,1,1,1,1,0,0). The closest reference I'm aware of is "A neural network approach to ordinal regression".

PS: If they can assume arbitrarily high values, it depends on the problem and goals at hand. For example, they can be represented as one variable with an accordingly adjusted cost function.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.