5
$\begingroup$

In context of regression, when does it not make sense to standardize a variable?

I understand that in binary & categorical variables, the mean and standard deviation are meaningless.

My understanding of the justification when appropriate is that 1/ it can reduce correlation between features, 2/ the effect of predictor variables on response variables will be on the same scale (standard deviations), enabling "apples to apples" comparison, 3/ sometimes a 1-unit increase in a variable is difficult to interpret whereas distance from the mean is inherently relative and 4/ the effect is assumed to be 0 at the predictor variable's mean.

Great! But is it applicable for all continuous distributions? What about...

  1. Skewed distributions
  2. Incomplete domains, not (-inf, inf) ex: Exponential

I understand that due to the central limit theorem, the sampling distribution of the mean of (virtually) any distribution will be normally distributed. So in context of regression when we're making statement about the mean, not specific units, perhaps CLT applies and so standardization is generally permissible.

This related question was not focused enough, yet given 3 upvotes, I'm hoping the increased granularity of my question will draw some answers.

I know that Cauchy is no go here due to infinite variance. Tbh I'm not sure how that works but for at least one distribution, the answer is "no".

Improve this question
$\endgroup$
2
  • $\begingroup$ I don’t follow what you’re saying about a Cauchy distribution. $X_1,\dots, X_n\overset{iid}{\sim}\text{Cauchy}$ have a finite empirical variance you would use in the standardization calculation. $\endgroup$
    – Dave
    Commented Jun 1 at 19:58
  • $\begingroup$ For skewed distributions you could apply transformations on the data like logistic or $\sqrt{}$ for example $\endgroup$
    – computer_goblin
    Commented Jun 2 at 0:27

1 Answer 1

Reset to default
7
$\begingroup$

All four of the reasons you give are, while not wrong, at least subject to debate and discussion.

  1. Lessen correlation among variables. First, it's colinearity and not regression you should be concerned with, and only if it's pretty high. And, if there are no outliers and the distribution isn't skewed, I'm not sure it does this. Can you give an example where it does?

  2. It's true that they will all be on the scale of standard deviation, but that is particular to your data set. If a variable has meaning (say, it's a length of something) then it stays the same from data set to data set. SDs do not.

  3. Sometimes that's true. And sometimes the opposite is true. In my experience, it's more often the opposite that's true, but I suppose if your scale is not well known, you don't lose much. But "the risk of cancer increases by 4.2% for each year over the age of 70" (just to make something up) is a lot more meaningful than the same thing with "standard deviation of age".

  4. This is true, since the mean will now be 0. But .... so? Is that desirable? I'm not saying it's bad, but why is it good?

So, rather than rules, I think you have to take it on a case by case basis. Sometimes, standardizing is good. Sometimes it is bad. Sometimes, it doesn't make too much difference. People vary on how often they think it is good --- I am more towards leaving the variables as is, but I recognize that there are times when standardizing can be good.

Improve this answer
$\endgroup$
3
  • $\begingroup$ For [4], I think it just makes it slightly easier to extract an effect of interest. Suppose you randomize a population, where all covariates are continuous, and wish to perform ANCOVA. If all covariates are standardized, then $\beta_{treatment}$ is already the treatment effect (on predictor variable), if I'm not mistaken (and maybe I am!) In other words, you wouldn't need $\beta_i * \mu_i$ for any of the covariates. $\endgroup$
    – jbuddy_13
    Commented Jun 1 at 22:30
  • $\begingroup$ The gist that I'm getting from your answer is that standardization is almost always an option but it's never required. Fair? $\endgroup$
    – jbuddy_13
    Commented Jun 1 at 22:41
  • 3
    $\begingroup$ I agree with that, but my main point is that you have to understand the pros and cons and then make a sensible decision $\endgroup$
    – Peter Flom
    Commented Jun 2 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.