2
$\begingroup$

I have a multiple linear regression model with several independent variables in different units. Because some of my data is negative, I am unable to take the log and therefore am standardizing the independent variables, by subtracting the mean and then dividing it by two standard deviations. Should I do this for all variables including dummy variables?

$\endgroup$
2
  • $\begingroup$ I think standardisation more commonly subtracts the mean and then divides by the standard deviation rather than by twice the standard deviation (leaving something with mean $0$ and variance $1$ rather than $\frac14$) $\endgroup$
    – Henry
    Jun 21 '19 at 14:41
  • $\begingroup$ See also stats.stackexchange.com/questions/359015/… $\endgroup$ Jun 24 '19 at 7:53
7
$\begingroup$

By dummy variables, I assume you mean dummy-coded categorical variables?

If so, then you do not need to standardize those. They only have two values, 0 (absence of thing) and 1 (presence of thing) so you can think of them as already standardized to 0=absence of thing.

In general, you should not standardize categorical variables, because they don't have an average for you to subtract. The numbers assigned to categories are arbitrary, so averaging them doesn't make sense. For example, suppose I have a sample of animals with a categorical variable for species, where 1=dog, 2=cat, 3=bird, 4=fish. In my sample there are 5 dogs, 2 cats, 2 birds and a fish, so my "average species" is 1.9 which doesn't really make sense.*

When you dummy-code the species variable, it turns into three variables that you can think of as is_dog, is_cat, and is_bird, each with no=0 and yes=1. There is no is_fish variable because I made fish my reference category (I know you're a fish if you're not a dog, a cat, or a bird).

So each dummy variable is automatically standardized with 0 meaning an observation is not of that species, and the coefficient for each tells you what you get when you change that value from 0 to 1. Overall, my species can be thought of as standardized to "fish", because when I model an observation where is_dog, is_cat, and is_bird are all 0, it's my reference category (fish), and when I model an observation where any of those is equal to 1 I am estimating the difference relative to fish (my beta for is_dog tells me how much higher/lower I expect my dependent variable to be for a dog than a fish).

If I were to change my reference category to cat, then my beta for is_dog would change, because now it tells me the difference between a dog and a cat instead of the difference between a dog and a fish. I would also need to add an is_fish dummy variable (coefficient tells me the difference between a fish and a cat) and get rid of the is_cat dummy variable (since cat is the reference, so there is now no difference between cat and reference). So you could then say that my model treating cat as reference might have species standardized to cat, since coefficients for all other species are given relative to cat.

* A very cat-like dog? Maybe it's a Chihuahua... but then if I subtract Chihuahua from bird to standardize my birds I get 3 - 1.9 = 1.1, so I guess a standardized bird is a slightly cat-like dog like a Doberman?

$\endgroup$
1
$\begingroup$

You don't have to standardize in a linear regression because the coefficient estimates take care of differences in units of the independent variables.

$\endgroup$
4
  • $\begingroup$ I based it off this previous question: stats.stackexchange.com/questions/100694/… ; because my coefficient estimates were huge before, I standardized them, basing it off an example in Gelman's book Data Analysis Using Regression and Multilevel/Hierarchical Models : lm(formula = kid.score ~ z.mom.hs + z.mom.iq + z.mom.hs:z.mom.iq) $\endgroup$
    – gisnewbie
    Jun 21 '19 at 4:45
  • $\begingroup$ Yes. I agree with the top answer to that question: sometimes standardization may help make the interpretation more convenient. But this point doesn't apply to dummy variables. $\endgroup$ Jun 21 '19 at 4:53
  • $\begingroup$ If I have other non-dummy variables, does that mean I should standardize those (just not the dummy variables)? $\endgroup$
    – gisnewbie
    Jun 21 '19 at 4:57
  • $\begingroup$ yes, if it helps with interpretation. $\endgroup$ Jun 21 '19 at 5:16
1
$\begingroup$

Some good answers have been posted to date. Here I focus on a bundle of misunderstandings within

Because some of my data is negative, I am unable to take the log and therefore am standardizing the independent variables

Wishing to take logarithms should imply that such a transformation is a good idea because of (say) nonlinear relationships, unequal variability, skewness of marginal distributions, or a desire to dampen the effects of outliers. What is key here is that standardization of any kind is a linear transformation and will not solve, or even improve, any of those problems.

More detail is needed on why you think you need a transformation. In particular, let's debunk a common myth: marginal distributions do not have to be normal for regression to work well. If that were true, then using indicator variables (you say dummy variables) would be completely out of order, as no indicator distribution can be anything but two spikes (or one spike ...). It's not essential to have approximately symmetric distributions either, although marked skewness often goes together with other problems.

But negative values of a predictor $x$ can be accommodated by transformations such as

  • cube root, calculated carefully as $x^{1/3}$ if $x \ge 0$ and $-(-x)^{1/3}$ if $x < 0$ or $\text{sign}(x) \ |x|^{1/3}$

  • "neglog" or $\text{sign}(x) \ln(1 + |x|)$

  • inverse hyperbolic sine (asinh)

and naturally the same transformations are sometimes helpful for a response too. As said, whether any of these is a good idea for your problem and data is an open question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.