Until now, I always normalized or standardized my features individually before feeding them into a neural network. But at my current project I have features, which in huge parts have the same unit (US-Dollars) and the neural network should basically find meaningful relations between those features (e.g. forming unknown ratios). Scaling the features individually would be therefore very harmful, because the data would lose the same unit which is important to find meaningful relations. Do you therefore agree that normalization is a bad idea in this case?
What would you suggest to conquer the problems arising from unnormalized data instead? Is there anything sensible I can do with e.g. the parameter initialization instead? Some features range from -10 to +10 and some range from -1000000 + 1000000. Anything else I should consider when working with such heterogenous data in Multi-Layer-Perceptrons? (activation function, optimizer...) Or do you think I can go without any normalization or anything else with this kind of data and still reach a meaningful result?
A related question was asked and answered here. The answer was essentially that normalization can be harmful if it is applied to data which has a common unit but it was not further explained what one should do in such a case.
(I'm currently building the network keras/theano.)