0
$\begingroup$

Until now, I always normalized or standardized my features individually before feeding them into a neural network. But at my current project I have features, which in huge parts have the same unit (US-Dollars) and the neural network should basically find meaningful relations between those features (e.g. forming unknown ratios). Scaling the features individually would be therefore very harmful, because the data would lose the same unit which is important to find meaningful relations. Do you therefore agree that normalization is a bad idea in this case?

What would you suggest to conquer the problems arising from unnormalized data instead? Is there anything sensible I can do with e.g. the parameter initialization instead? Some features range from -10 to +10 and some range from -1000000 + 1000000. Anything else I should consider when working with such heterogenous data in Multi-Layer-Perceptrons? (activation function, optimizer...) Or do you think I can go without any normalization or anything else with this kind of data and still reach a meaningful result?

A related question was asked and answered here. The answer was essentially that normalization can be harmful if it is applied to data which has a common unit but it was not further explained what one should do in such a case.

(I'm currently building the network keras/theano.)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

For an example such as the one you have given above you can try experimenting with something less than full normalization, but more than just the raw input data. Using your example where you have variables all in USD that say range from:

  • Variable A: -100 to 100 USD
  • Variable B: -100000 to 100000 USD
  • Variable C: -1000000 to 1000000 USD

all of which might have equal significance in reality, you could change the units to something meaningful to you for individual variable that will make them the same magnitude. For example:

  • Variable A: -1 to 1 USD, hundreds
  • Variable B: -1 to 1 USD, hundred thousands
  • Variable C: -1 to 1 USD, millions

By doing so you have kept the units in something that you can understand, but you have also made it far easier for most algorithms to handle.

Improve this answer
$\endgroup$
2
$\begingroup$

The problem with not transforming the input features is that the ANN may not even learn what it is supposed to, and the huge correlation between features (with large scale values of -1e6 and 1e6) will swamp the learning process. You have to tone down the range to [-1,1] and almost always decorrelate the features prior to input, since an ANN will waste time learning the correlation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ About the necessity of decorrelation I found mixed answers in studies and on the internet. But back to the core question: Am I right that you are referring to the [-1,1] range because you are assuming a sigmoid activation function in the first layer? Am I a bit more flexible, if I'm using a relu function? Is the following a good idea(?): Refactor the features e.g. by dividing all input values of a feature by it's mean of the absolute values. This would not exactly bring the values in the [-1,1] range, but tone down the range significantly but still preserve the ratios (just refactored) $\endgroup$
    – SebastianB
    Mar 10, 2017 at 18:25
  • $\begingroup$ That should help. $\endgroup$
    – user32398
    Mar 11, 2017 at 20:51
  • $\begingroup$ The point for [-1,1] is merely a long-standing recommendation for uniformity (in scale) of ANN input features. Sure the is normalization, as well as mean-zero-standardization, as well as what activation function you use. However, if there are outlier observations whose values are more than 2 s.d. from the features's mean, you will be throwing off the ANN again. Normalization can pass through outlier influences as well, so you have to consider percentiles in that case. The decorrelation is a learning issue. You're on the right track. Look at the DDR package at jurikres.com $\endgroup$
    – user32398
    Jun 6, 2018 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.