5
$\begingroup$

I want to check the assumptions for applying linear regression analysis. So, among others I check the linear dependency between my dependent (which is continuous) and my independent (nominal or dummy) variables.

As scatterplots and Pearson or Spearman correlations are not the right measure to check the linearity assumption in my case, I wonder what is another useful way applicable in my case with a continuous dependent and nominal or dummy independent variables?

Thank you for your help!

$\endgroup$
  • 2
    $\begingroup$ If your only independent variables are nominal, there is no meaningful linearity assumption to check. $\endgroup$ – gung - Reinstate Monica Nov 18 '14 at 20:55
  • $\begingroup$ So linear regression is not a good idea with this data? $\endgroup$ – jeffrey Nov 18 '14 at 20:56
  • 3
    $\begingroup$ Linear regression is fine. The linearity assumption cannot possibly be violated when you only have nominal independent variables. $\endgroup$ – gung - Reinstate Monica Nov 18 '14 at 21:03
  • $\begingroup$ Thank you. Could you tell me why the lin. assumption cannot be violated? $\endgroup$ – jeffrey Nov 18 '14 at 21:05
  • 1
    $\begingroup$ I have an answer somewhere that talks about this. If I can find it, I'll link to it later. @Scortchi's answer below gives you the idea with a single dummy variable. It is a straightforward generalization from there to cases w/ more. $\endgroup$ – gung - Reinstate Monica Nov 18 '14 at 21:15
8
$\begingroup$

Let me explain what linearity means with nominal/dummy variables. In essence, it means there is no interaction term between your independent variables that you have left out.

Suppose we have two nominal variables $x_0$ and $x_1$, each taking values 0 or 1, and a response variable $y$. (The general case is similar.)

If we model $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \epsilon$:

$\beta_0$ is the expected response when $x_1 = x_2 = 0$

$\beta_0 + \beta_1$ is the expected response when $x_1 = 1, x_2 = 0$

$\beta_0 + \beta_2$ is the expected response when $x_1 = 0, x_2 = 1$

$\beta_0 + \beta_1 + \beta_2$ is the expected response when $x_1 = x_2 = 1$

There's a relationship here, since we have 3 coefficients but four cases: The last minus the first is the sum of the second minus the first and the third minus the first.

If this relationship actually holds in your situation between the expected responses, then this linear model can be a good one. If not, then the failure of this relationship is a type of nonlinearity.


If we include an interaction term, then linearity is automatically satisfied, because we have four coefficients to fit the four cases. That is, with a model $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_1 x_2 + \epsilon$ there is no restriction on the relationship between the expected responses in the four cases above. (However the distributions of y in these 4 cases may still be different, which would violate the model as written.)


How do you test whether you can leave out the interaction term? One way would be to try including it and test whether the coefficient $\beta_3$ is distinct from zero. For example, in the case of normal error $\epsilon$, this would be a $t$-test for a slope coefficient in a regression.


† An interaction between $x_1$ & $x_2$ is a type of (multi-dimensional) nonlinearity: there's no possibility of a nonlinear relationship between $\operatorname{E}Y$ and $x_1$ when $x_1$ is a dummy variable, but there is between $\operatorname{E}Y$ and $(x_1,x_2)$. That is, there may be no plane passing through the four points $(0,0,\operatorname{E}(Y|\,0,0))$, $(1,0,\operatorname{\operatorname{E}}(Y|\,1,0))$, $(0,1,\operatorname{E}(Y|\,0,1))$, $(1,1,\operatorname{E}(Y|\,1,1))$.

For dummy variables, these interaction terms are the only potential source of nonlinearity of the expected responses.

$\endgroup$
  • 2
    $\begingroup$ I personally wouldn't call an interaction a non-linearity, but this is a useful contribution to the thread, +1. Welcome to CV, this is well-thought through, & you've mastered our formatting options, too. I hope we'll see more in the future. $\endgroup$ – gung - Reinstate Monica Nov 18 '14 at 22:08
  • 1
    $\begingroup$ It's a very clear exposition of interactions between categorical variables. Nevertheless, I'm not at all sure calling an interaction a non-linearity is a matter of personal preference - for a given value of $x_2$ there's still no question of a linear or curvilinear relationship between $\operatorname{E}Y$ & $x_1$ - does anyone else do that? $\endgroup$ – Scortchi - Reinstate Monica Nov 18 '14 at 22:24
  • 1
    $\begingroup$ @Scortchi There's no possibility of a nonlinear relationship between EY and $x_1$ as you say, but there is between EY and $(x_1,x_2)$. That is, there may be no plane passing through the four points $(0,0,E(Y|0,0)), (1,0,E(Y|1,0)), (0,1,E(Y|0,1)), (1,1,E(Y|1,1))$. I must say I'm a mathematician and not a statistician, so I don't necessarily know what statisticians mean when they use the term, but to my thinking, an interaction is a type of (multi-dimensional) nonlinearity. $\endgroup$ – awcc Nov 18 '14 at 22:35
  • 3
    $\begingroup$ @gung: Perhaps the clearer term to use would be non-additivity. In a sense, it is additivity that is being violated with continuous predictors when there is a non-linear mean function: $E(Y|X=2x) = E(Y | X=x+x) \neq E(Y|X=x) +E(Y|X=x)$ when such non-linearity exists. $\endgroup$ – Andrew M Nov 19 '14 at 8:12
  • 1
    $\begingroup$ @AndrewM, that's a useful way of moving up in abstraction to the more general case. $\endgroup$ – gung - Reinstate Monica Nov 19 '14 at 14:20
7
$\begingroup$

Consider the model for regression of $y$ on $x$: $$\operatorname{E}Y=\beta_0 + \beta_1 x$$ If $x$ is a dummy variable representing a level of a categorical variable then it can take only the values 0 & 1: $$\operatorname{E}Y=\beta_0 \qquad \text{when } x=0$$ $$\operatorname{E}Y=\beta_0 + \beta_1 \qquad \text{when } x=1$$ and there is no question of linearity, merely two different fitted responses for each of two different groups. (Two points always lie in a straight line.)

You can have other predictors $\boldsymbol{x}$ in the model $$\operatorname{E}Y=\beta_0 + \beta_1 x_1 + \boldsymbol\beta^\mathrm{T}\boldsymbol{x}$$

; for any given values of those there are still only two values of $x_1$ to compare, & the difference in the response is given by $\beta_1$

$$\operatorname{E}Y=\beta_0 + \boldsymbol\beta^\mathrm{T}\boldsymbol{x}\qquad \text{when } x_1=0$$ $$\operatorname{E}Y=\beta_0 + \beta_1 + \boldsymbol\beta^\mathrm{T}\boldsymbol{x}\qquad \text{when } x_1=1$$

As @awcc explains, interaction can be considered a form of non-linearity, but it's not what people usually mean when they talk about non-linearity in the context of regression.

$\endgroup$
  • $\begingroup$ Thank you! Does this also hold, when I have a mixture of several dummies and also continuous variables as X's ? $\endgroup$ – jeffrey Nov 18 '14 at 21:16
  • 1
    $\begingroup$ Yes (for the dummies). As gung @says, it's a straightforward generalization; if he doesn't find his answer I'll illustrate it. $\endgroup$ – Scortchi - Reinstate Monica Nov 18 '14 at 21:20
  • $\begingroup$ @jeffrey, the relationship b/t the continuous variables & Y can be curvilinear. Sorry, but I'm not finding my answer (which probably wasn't as good as what Scortchi might offer anyway). $\endgroup$ – gung - Reinstate Monica Nov 18 '14 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.