My neural network can have all sorts of inputs from different datasets. For example, with digit recognition using the MNIST dataset, there are 784 inputs (each pixel 28x28) and each value is between 0-255 (single grayscale). However, this would produce math range errors with the sigmoid function because too larger negative values would be produced on later layers. So, all value are divided by 255 to get decimal values between 0-1. Is that correct?

Well, does this mean with other datasets, that have big values, you can just divide them all by 10 or 100 to make them smaller? Does that matter? What is this process known as?


This process is known as "normalization" or "transformation" and is part of your feature engineering. Your problem applies to all machine learning algorithms, not just neural networks.

We usually prefer [0,1], because they are easier to deal with. There's no rule on what and how to normalize, but you should think:

  • Are my variables on a comparable scale?
  • Does my machine learning require normalization?
  • Is my variable discrete, should I transform it to continuous?

You certainly shouldn't just divide your variables randomly to make it smaller. In your image classification, dividing by 255 is good because the whole range is in [0,1]. You can't have anything less than 0 and greater than 1.

  • $\begingroup$ Would it make sense to divide each row by its own maximum then? $\endgroup$ Dec 25 '16 at 12:33
  • $\begingroup$ @harrylakins No. Your maximum is only the max in your sample, and might not have much to do with your population. Also, in many problems there is no maximum value. $\endgroup$
    – SmallChess
    Dec 25 '16 at 12:34
  • $\begingroup$ Ah yes - so how do you achieve 0-1 normalisation? $\endgroup$ Dec 25 '16 at 12:35
  • $\begingroup$ @harrylakins It depends on your problem. In your image classification, you divide by 255 because there are 255 values. But if your colour map has more than 255 values, your would need to divide with something more than 255. $\endgroup$
    – SmallChess
    Dec 25 '16 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.