57
$\begingroup$

In what circumstances would you want to, or not want to scale or standardize a variable prior to model fitting? And what are the advantages / disadvantages of scaling a variable?

$\endgroup$
37
$\begingroup$

Standardization is all about the weights of different variables for the model. If you do the standardisation "only" for the sake of numerical stability, there may be transformations that yield very similar numerical properties but different physical meaning that could be much more appropriate for the interpretation. The same is true for centering, which is usually part of the standardization.

Situations where you probably want to standardize:

  • the variables are different physical quantities
  • and the numeric values are on very different scales of magnitude
  • and there is no "external" knowledge that the variables with high (numeric) variation should be considered more important.

Situations where you may not want to standardize:

  • if the variables are the same physical quantity, and are (roughly) of the same magnitude, e.g.
    • relative concentrations of different chemical species
    • absorbances at different wavelengths
    • emission intensity (otherwise same measurement conditions) at different wavelengths
  • you definitively do not want to standardize variables that do not change between the samples (baseline channels) - you'd just blow up measurement noise (you may want to exclude them from the model instead)
  • if you have such physically related variables, your measurement noise may be roughly the same for all variables, but the signal intensity varies much more. I.e. variables with low values have higher relative noise. Standardizing would blow up the noise. In other words, you may have to decide whether you want relative or absolute noise to be standardized.
  • There may be physically meaningful values that you can use to relate your measured value to, e.g. instead of transmitted intensity use percent of transmitted intensity (transmittance T).

You may do something "in between", and transform the variables or choose the unit so that the new variables still have physical meaning but the variation in the numerical value is not that different, e.g.

  • if you work with mice, use body weight g and length in cm (expected range of variation about 5 for both) instead of the base units kg and m (expected range of variation 0.005 kg and 0.05 m - one order of magnitude different).
  • for the transmittance T above, you may consider using the absorbance $A = -log_{10} T$

Similar for centering:

  • There may be (physically/chemically/biologically/...) meaningful baseline values available (e.g. controls, blinds, etc.)
  • Is the mean actually meaningful? (The average human has one ovary and one testicle)
$\endgroup$
  • $\begingroup$ +1 and accepted because of the helpful list of when to and when not too standardize, thanks $\endgroup$ – Andrew Dec 14 '11 at 16:02
  • 6
    $\begingroup$ +1 for "The average human has one ovary and one testicle" (& also for the rest of the answer ;-). $\endgroup$ – gung - Reinstate Monica Dec 13 '12 at 14:35
  • 1
    $\begingroup$ @cbeleites is there any chance you could provide a link to a resource that explains baseline channels in the context you've used in your answer? I have not heard the term before and I'm getting search results which are not helpful in understanding your use of the term here. Thanks! $\endgroup$ – mahonya Dec 7 '13 at 16:15
  • 1
    $\begingroup$ @sarikan: have a look at fig. 1 in this article: americanlaboratory.com/913-Technical-Articles/… for biological and physico-chemical reasons, in the range between 2000 and 2700 cm$^{-1}$ no signals are expected. This region may be used to estimate the baseline (from physical effects that are not Raman) which is then subtracted. These variates will then be approximately zero plus some noise. $\endgroup$ – cbeleites supports Monica Dec 7 '13 at 21:32
9
$\begingroup$

One thing I always ask myself before standardizing is, "How will I interpret the output?" If there is a way to analyze data without transformation, this may well be preferable purely from an interpretation standpoint.

$\endgroup$
7
$\begingroup$

In general I don't recommend scaling or standardization unless it's absolutely necessary. The advantage or appeal of such a process is that, when an explanatory variable has a totally different physical dimension and magnitude from the response variable, scaling through division by standard deviation may help in terms of numerical stability, and enables one to compare effects across multiple explanatory variables. With the most common standardization, the variable effect is the amount of change in the response variable when the explanatory variable increases by one standard deviation; it also indicates that the meaning of the variable effect (the amount of change in the response variable when the explanatory variable increases by one unit) would be lost although the statistical value for the explanatory variable remains unchanged. However, when interaction is considered in a model, scaling could be very problematic even for statistical testing because of a complication involving a stochastic scaling adjustment in calculating the standard error of the interaction effect (Preacher, 2003). For this reason, scaling by standard deviation (or standardization/normalization) is generally not recommended, especially when interactions are involved.

Preacher, K.J., Curran, P.J., and Bauer, D. J., 2006. Computational tools for probing interaction effects in multiple linear regression, multilevel modeling, and latent curve analysis. Journal of Educational and Behavioral Statistics, 31(4), 437-448.

$\endgroup$
  • 4
    $\begingroup$ I question your claim that standardizing predictors is "generally not recommended, especially when interactions are involved." Neither Gelman and Hill, nor Raudenbush & Bryk mention this concern in their texts. But when I have a chance I will look at the references you mention with interest. $\endgroup$ – Michael Bishop Dec 1 '11 at 20:16
  • $\begingroup$ If we use the calibration universe std as the scaling variable, then the scaling is not stochastic. $\endgroup$ – adam Jan 28 '15 at 21:33
  • $\begingroup$ Can someone confirm if scaling is harmful in case of interaction terms? That does not seem to have been resolved in the above discussion. $\endgroup$ – Talik3233 Jun 30 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.