4
$\begingroup$

I am using an OLS model to predict a continuous variable using several continuous predictors and one binary categorical predictor. I know that usually binary variables are modeled as [0, 1], but I am curious what the effect of giving a binary categorical predictor values of [-1, 1] would be. I had accidentally done the [-1, 1] coding in the first round of analysis, and after correcting the coding to [0, 1] I saw that the change in coding made the regression result similar but less extreme. I realize I don't fully understand what change this had on the model, and I'd appreciate insight into why it is so important to code the categorical variable as [0, 1].

[Edit] To clarify what I mean by the result being 'less extreme': I am fitting this model separately for each individual in our dataset, and then I do a wilcoxon signed-rank test across the coefficients from the binary variable (the coefficients do not meet the normality assumption for a t-test). The test is significant for the coefficients resulting from the [-1, 1] coding but not for the [0, 1] coding. Naturally I would like the test to be significant but now I feel that I cannot accept this result because it is arbitrarily affected by the coding. Is testing the coefficients resulting from the [-1, 1] coding somehow more or less legitimate? I am not sure which test result to accept, or whether I have approached the problem incorrectly.

$\endgroup$
7
  • 5
    $\begingroup$ In what sense where the results less extreme? You should hage got equivalent results, and there is no sense in which it is important to code as 0,1 other codings could potentially be useful as well $\endgroup$ Feb 23 at 14:54
  • 5
    $\begingroup$ The only difference concerns how you interpret the estimates in its output. The coding does not affect the fit or its predictions in any way. Thus, there must be something arbitrary about how you assess whether the result is "extreme." $\endgroup$
    – whuber
    Feb 23 at 15:32
  • 5
    $\begingroup$ This is a bit like switching measurement from centimeters to meters and saying your effects are less extreme because they're 100 times smaller. If you take into account your change of units, everything is entirely the same. What you've done is similar to switching from measuring the dependent variable in pints to measuring in quarts. There are two pints in a quart and your effect sizes will be half as big (your intercept will be affected by a slightly more complicated affine transformation). $\endgroup$ Feb 23 at 16:25
  • $\begingroup$ Thanks all for your very helpful advice! I have a better understanding now of how the coding affects the model - however, I am still a bit confused about why tests of the coefficients are affected by the coding (see edit above where I clarify what I mean by 'more extreme'). $\endgroup$
    – j45612
    Feb 24 at 12:07
  • 1
    $\begingroup$ " a vector containing the coefficients from all individuals" can you explain more about this. How do you get that vector, and what does it represent? How is that vector a paired sample; do you have two vectors, and if so, how are they obtained? How is the binary predictor involved? Please explain the things that might be obvious to you but are unclear to the readers of your question. $\endgroup$ Feb 24 at 13:39

3 Answers 3

4
$\begingroup$

If the model with $X$ coded as $\{0,1\}$ has the following form: $$ y_i = \alpha_0 + \alpha_1 X_i + \varepsilon $$ and model with $X^*$ coded as $\{-1, 1\}$ has the following form: $$ y_i = \beta_0 + \beta_1 X^*_i + \varepsilon $$ we can write the following equivalences:

$$ \beta_0 = \alpha_0 + \frac{\alpha_1}{2} \\ \beta_1 = \frac{\alpha_1}{2} $$

$\alpha_0$ is the predicted mean of the outcome when $X=0$, and $\alpha_1$ is the difference between the predicted mean of the when $X=1$ and the predicted mean when $X=0$, i.e., the difference between the two predicted means. $\beta_0$ is the midpoint between the two predicted means, and $\beta_1$ is the difference between the predicted mean when $X=1$ and the midpoint. When other predictors are in the model, these interpretations involve holding those other variables constant (i.e., for a pair of units that were the same on all other covariates but differed in $X$).

The p-value on the test for $\beta_1$ will be identical to that for $\alpha_1$, but this will not be true for $\beta_0$ and $\alpha_0$.

$\endgroup$
2
  • $\begingroup$ "The p-value on the test" the use of 'the test' might need some extra specification. With a t-test the statement is correct, But with ANOVA it will depend on which type of variances are being compared. $\endgroup$ Feb 24 at 12:22
  • $\begingroup$ Can you elaborate your logic for being able to write "the following equivalences" a little? It seems to have cost you far less effort to see this than I care to admit for my rather more clumsy argument. $\endgroup$ Feb 25 at 9:28
2
$\begingroup$

The operation of transforming a $[0,1]$ dummy into a $[-1,1]$ dummy amounts to (see below for a more general case) transforming the categorical variable (it does not matter that it is categorical) via $2x-1$ and leaving the constant as is.

Via Scaling in linear regression and Adding a Constant to Every Column of X (OLS), this means first scaling the regressor matrix via $$ \begin{pmatrix} 1&0\\0&2 \end{pmatrix} $$ to then add $$ \begin{pmatrix} 0&-1\\ \vdots&\vdots\\0&-1 \end{pmatrix} $$ to the transformed regressor matrix. Now, this is, from the second link provided above, the same as multiplying the regressor matrix with $$ \begin{pmatrix} 1&-1\\0&1 \end{pmatrix} $$ Hence, we transform the originial regressor matrix with $$ \begin{pmatrix} 1&0\\0&2 \end{pmatrix} \begin{pmatrix} 1&-1\\0&1 \end{pmatrix}=\begin{pmatrix} 1&-1\\0&2 \end{pmatrix}=:A $$ From the first link, the new and old coefficient vectors are related via $$ \hat{\beta}^\circ=A^{-1}\hat{\beta}, $$ where $$ A^{-1}=\begin{pmatrix} 1&0.5\\0&0.5 \end{pmatrix} $$ All in all, the new coefficient for the constant will then be the old constant plus one half the old slope coefficient and the new slope will be one half the old slope. This is also intuitive, as moving from the lowest to the highest value of the regressor now is twice as far, so that increasing by one unit should only be half the effect.

Numerical illustration:

n <- 20

y <- rnorm(n)
x <- sample(c(0,1), n, replace = T) # a categorical predictor...
x <- rnorm(n)                       # ...but works for any predictor
x.trafo <- 2*x-1

> all.equal(0.5*coef(lm(y~x))[2], coef(lm(y~x.trafo))[2], check.attributes = F)
[1] TRUE

> all.equal(coef(lm(y~x))[1] + 0.5*coef(lm(y~x))[2], coef(lm(y~x.trafo))[1], check.attributes = F)
[1] TRUE

A more general case

Consider a constant and regressors $x_1,\ldots,x_k$, scaling factors to all variables, $s_0,s_1,\ldots,s_k$ and additions to regressors $\mathbf{a}:=(a_1,\ldots,a_k)'$ (since scaling and adding is equivalent for the constant, we omit $a_0$). Let $\mathbf{s}:=(s_1,\ldots,s_k)'$.

Then, the operation of scaling and adding can be expressed via a matrix $$ S:=\begin{pmatrix} s_0&a_1&\cdots&\cdots&\cdots&a_k\\ 0&s_1&0&\cdots&\cdots&0\\ 0&0&s_2&\ddots&\cdots&0\\ 0&\vdots&\ddots&\ddots&\ddots&0\\ 0&0&&&&0\\ 0&0&0&\cdots&0&s_k\\ \end{pmatrix}=\begin{pmatrix} s_0&\mathbf{a}'\\ \mathbf{0}&\text{diag}(\mathbf{s})\\ \end{pmatrix} $$ This representation also reveals that the $s_j$ must not be zero. In the case of $s_0$ we would create a zero column. For $s_j$, $j\neq0$, the a corresponding $a_j\neq0$ would ensure a nonzero column for the transformed regressor even if $s_j=0$, but then, that transformed regressor would simply be $a_j$ and hence be collinear to the constant $s_0$.

By partitioned inverses, $$ S^{-1}=\begin{pmatrix} 1/s_0&-\mathbf{a}'\text{diag}(\mathbf{s})^{-1}/s_0\\ \mathbf{0}&\text{diag}(\mathbf{s})^{-1}\\ \end{pmatrix} $$ and the coefficients in the transformed dataset, $\hat{\beta}^\circ$, are related to those of the original one, $\hat{\beta}$, via $$ \hat{\beta}^\circ=S^{-1}\hat{\beta} $$ Hence, as also pointed out by Matthew in the comments, the new slope coefficients are simply the old ones reciprocally adjusted for by their own respective scaling while the new intercept is a function of all additions and all scalings.

$\endgroup$
1
$\begingroup$

Let me provide an explanation for this query with a simpler scenario. Let's just take one scalar variable as the response variable, and one dichotomous variable as the predictor variable. If you run OLS regression, you actually are just doing a 2 independent sample t-test (assuming homogeneity of variances between the two groups). The key is to recognize that the dichotomous variable really is just a group-membership indicator variable.

Now, the results for the null hypothesis statistical test (NHST) are going to be the same as the results for the NHST asking if the slope of the OLS regression is different from zero. (The t-ratio and p-values will be the same.)

So, here's the key to the OLS approach. If you run the regression, you get something like $$y = 12.34 + 5.67 D$$ (extracting the slope and intercept from the OLS output). Here, I've used $D$ as the predictor variable (instead of $x$) because I want to indicate that it is a dichotomous variable (takes on only 1 of 2 values).

So, ¿what are the slope and intercept? This is where the coding of {0,1} or {-1,1} comes into play. If you use the former coding scheme of {0,1}, then the intercept of 12.34 is the mean of the $D=0$ group, and the slope is the difference in the group means. So, if you plug in $D=1$, you can get the other group's mean from this equation: 12.34 + 5.67 = 18.01.

Now, if you use the latter coding scheme of {-1,1}, then the intercept is the GRAND mean (the mean if you put both data sets together). In this case, the slope is half the difference between the group. So, to get each of the groups means, you need to plug in the value for $D$ for each group: $D=-1$ would give a group mean of $12.34 - 5.67 = 6.67$ and $D=+1$ would give a group mean of $122.34 + 5.67 =18.01$.

Note, I'm assuming the same number of members in each group...but the ideas essentially can be adjusted to extend to account for unbalanced groupings.

The idea extends to more complicated models with other scalar predictors.

I hope this is helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.