4
$\begingroup$

I have found that for some datasets, mean removal and variance scaling helps to fit a better model to data while for some datasets this does not help.

On what kind of data standardization will be helpful?

Are there some guidelines for applying this?

$\endgroup$
1

2 Answers 2

9
$\begingroup$

First off, standardization usually is taken to be

  1. subtraction of the mean

  2. division by the standard deviation.

The result has a mean 0 and standard deviation of 1.

Dividing by the variance will be wrong for any variable that is not a pure number. One of the reasons for standardization is to remove any influence of the units of measurement. The standard deviation always has the same units of measurement as the variable itself and division washes out those units.

There is no reason in principle why e.g. subtraction of the median and division by the interquartile range or in general any scaling

(value - measure of level) / measure of scale

might not be useful, but using mean and SD is by far the most common procedure. The idea that the Gaussian or normal is a reference distribution often underlies this, but using measures of level and scale other than the mean and standard deviation would often be useful, especially if you were interested in simple methods for identifying outliers (a very big topic covered by many threads in this forum).

The answer to your general question is pretty much tautologous: standardization is useful whenever difference in level, scale or units of measurement would obscure what you want to see. If you are interested in relative variations, standardize first.

If you wanted to compare the heights of mean and women, the units of measurement should be the same (metres or inches, whatever), and standardization is not required. But if the scientific or practical question requires comparing values relative to the mean, subtract the mean first. If it requires adjusting for different amounts of variability, divide by the standard deviation too.

Freedman, D., Pisani, R., Purves, R. Statistics New York: W.W. Norton (any edition) is good on this topic.

$\endgroup$
3
  • $\begingroup$ Standardising is also useful if you want to combine variables of different scales together to produce a summary measure - it gets them facing in the same direction. $\endgroup$
    – user20650
    Commented Jun 23, 2013 at 12:08
  • 1
    $\begingroup$ @user20650 Not so to "facing in the same direction". Standardization by mean and SD preserves correlations, positive and negative, or otherwise put, correlations are based on standardization. So standardization will change no signs of relationship. $\endgroup$
    – Nick Cox
    Commented Jun 23, 2013 at 12:11
  • 1
    $\begingroup$ In many cases, if standardizing improves the result, something is wrong with the overall model. It is usually better to model what needs to be standardized for rather than doing a two-step analysis. $\endgroup$ Commented Jul 7, 2013 at 15:16
2
$\begingroup$

+1 to @Nick Cox, who has provided you with a good answer. Let me supplement that briefly.

Whether centering or standardization leads to a "better model" will depend on what it means for one model to be better than another model. Here are some cases where it seems to me that one might consider a model to be "better", and thus might prefer to center or standardize:

  • If you center (which perforce includes standardizing) your variables before you create power terms or interaction terms for a multiple regression model, you will minimize the amount of collinearity that would otherwise have been created. This can aid in the numerical stability of the estimation algorithms, and the power and interpretability of the inferences.
  • When conducting analyses like Principal Components Analysis, Factor Analysis, or Cluster Analysis, differences in the variances of the different inputs can have strong effects on the output. This can be an issue if the various measurements are incommensurate. Consider a case where you have a number of variables, two of which are length and time. The universe doesn't care whether you measure things in centimeters or in kilometers, in seconds or in years; however, switching the units for one of your variables will yield a massively different number for the variance, which in turn will yield vastly different results from these kinds of analyses. Thus, it is often recommended that you standardize your variables before conducting these analyses.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.