3
$\begingroup$

I have had some success training my deep neural network (with ReLU hidden units) by first normalizing the features of my data set to zero-mean-unit-variance.

Each sample of my data set has 600+ values. Majority of them are continuous but a handful or columns are symbolic / discrete (represent by fixed range integers). Currently I just normalize all columns to zero-mean-unit-variance the same way. Should I only normalize the continuous value columns? Are there better data normalization strategies?

$\endgroup$
8
  • $\begingroup$ By continuous features do you mean something like spatial or temporal continuity? What type of data you are using? $\endgroup$ Commented Jun 15, 2015 at 13:06
  • $\begingroup$ They are financial models where numbers are percentage changes. $\endgroup$
    – teddy
    Commented Jun 15, 2015 at 14:56
  • $\begingroup$ If adjacent features are mostly correlated (or continuous), I think that is also your case, I don't recommend feature based normalization. Instead sample based normalization would be better. That is simply subracting mean of each sample from itself and also dividing of each sample's std to itself. I'm not familiar with data type you mention, If you think that will eliminate some important information from the data, don't apply any normalization on continuous part. $\endgroup$ Commented Jun 15, 2015 at 15:35
  • $\begingroup$ thanks. will try your suggestion of sample based normalization. not normalizing seems to leave poor results though. $\endgroup$
    – teddy
    Commented Jun 16, 2015 at 17:45
  • $\begingroup$ What do you mean by 'symbolic'? Are they categorical? If so, they should be converted to multiple binary variables. $\endgroup$
    – pir
    Commented Jun 17, 2015 at 7:21

1 Answer 1

5
$\begingroup$

You said your integer-valued variables have a fixed range, which makes things relatively easy. You will need to encode them as several features rather than one.

  • If they are categorical (there is no inherent natural order within them), you can use one-hot encoding (i.e. the "fifth option" is represented as (0,0,0,0,1,0,0)), this is the common practice.
  • If they are ordinal (discrete, but with an order), using one-hot is possible, but it's even better to use (1,1,1,1,1,0,0). The closest reference I'm aware of is "A neural network approach to ordinal regression".

PS: If they can assume arbitrarily high values, it depends on the problem and goals at hand. For example, they can be represented as one variable with an accordingly adjusted cost function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.