Often a preprocessing technique to do is to normalize our data in a range (0,1) before we tow our model (example neural network) on them.
I understand why in a practical way (for example if we have the height of something in kilometers and its weight in like picograms).
However, if we are dealing with, for example, images in a range (0,255) why do we convert these in a range (0,1) (through a images = images /255. ).
Everyone talks about the benefits of doing this, but I haven't found any papers that actually talk about it. Could you give me some papers/books where this is discussed?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ My answer below is very elementary compared with the threads declared as duplicate -- but few of the basic points it makes are covered in the duplicates. $\endgroup$– Nick CoxFeb 1, 2023 at 13:06
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is a good but very general question and I can't hope to cover all that might be valuable about such scaling, or problematic about it either. Indeed I am hopeful that others can remind me or inform me of points I don't cover, so I too can benefit.
But I think the question can be divided into
Why is a common range useful?
Why is 0 to 1 useful in particular?
Small print that can bite Usually, at least in what I see, the interval concerned is $[0, 1]$ so that $0$ and $1$ are, depending on context, attainable in principle, attained in practice, or both. The interval notation $(0, 1)$ is often used to indicate an interval exclusive of bounds so that $0$ and $1$ themselves are not allowed. Fields and even national traditions in different countries vary in what is customary notation and how strictly it is used.
Historically, scaling to similar if not identical ranges was important for numerical stability. This remains important but the difference now is that decent software will do this for you on the fly.
The importance of scaling to a common range is partly to ensure that we aren't distracted by (e.g.) different units of measurement or different characteristic magnitudes. Thus only a very naive person would consciously compare temperature on the Celsius scale with rainfall in mm or inches, but successful use of many techniques (particularly if there is an idea of comparing relative importance of variables in some sense) depends on not being distracted by matters like that.
Other way round, scientists in particular and many other researchers should have subject-matter knowledge that gives a good feeling for e.g. what change is associated with a rise of $1^\circ$ C or extra expenditure of USD $1$ million, or whatever.
The advantages of $[0, 1]$ can include simplicity and the fact that such numbers are moderate in size. A recipe (value $-$ min) / (max $-$ min) is easy to explain and implement. Sometimes the fact that such numbers may be regarded as proportions or probabilities may be helpful.
Everything depends on context so that someone used to image processing may be so accustomed to [0, 255] as the interval they work on that they find working with [0, 1] much harder -- but the opposite is also true!
There are other kinds of scaling in use such as standardizing in the sense (value $-$ mean) / SD which ensures that all variables have mean $0$ and SD $1$ (and thus also variance $1$), but doesn't ensure that all such scalings cover the same range.
More small print that can bite Never assume a particular meaning for scaling, normalizing or standardizing unless you can see the precise recipe used (even better, the code). These terms are not standardized (indeed) in meaning across statistics and machine learning, let alone science or mathematics.
