What's the difference between regression and analysis of variance?

Question

Like, I don't get it. In regression, we say $$Y_i = a + bX_i + \epsilon$$ where $X_i$ is just a real number for each observation $i$,

and in analysis of variance with two groups, we say $$Y_i = a + bX_i + \epsilon$$ where $X_i$ is again just a real number, but this time either 0 (group 1) or 1 (group 2), so a is mean of group 1, and a+b is mean of group 2.

But despite the interpretation, the model is exactly the same. So why do people split it up into regression and anova? the model is the same?

Answer 1

ANOVA isn't a model --- it is a method within a model

The analysis of variance (ANOVA) is a method that occurs within regression models. The technique is based on the law of iterated variance. Suppose you have some regression model:

$$Y_i = f(\mathbf{x}_i, \theta) + \varepsilon_i \quad \quad \quad \varepsilon_1,...,\varepsilon_n \sim \text{IID Dist}(0, \sigma^2).$$

Using the law of iterated variance we can write the marginal variance of $Y_i$ as:

$$\begin{equation} \begin{aligned} \mathbb{V}(Y_i) &= \mathbb{V}(\mathbb{E}(Y_i|\mathbf{X}_i)) + \mathbb{E}(\mathbb{V}(Y_i|\mathbf{X}_i)) \\[6pt] &= \mathbb{V}(f(\mathbf{X}_i, \theta)) + \mathbb{E}(\varepsilon_i) \\[6pt] &= \mathbb{V}(f(\mathbf{X}_i, \theta)) + \sigma^2. \\[6pt] \end{aligned} \end{equation}$$

Now, if the explanatory vector does not have a relationship with the response variable then the regression function does not depend on the explanatory vector, and so $\mathbb{V}(f(\mathbf{X}_i, \theta)) = 0$, which implies $\mathbb{V}(Y_i) = \sigma^2$. On the other hand, if the explanatory vector does have a relationship with the response variable, then we will generally have $\mathbb{V}(f(\mathbf{X}_i, \theta)) > 0$, which implies $\mathbb{V}(Y_i) > \sigma^2$. Thus, generally speaking, a larger gap between the estimated variance of the response variable, and the estimated variance of the error term, constitutes evidence in favour of the hypothesis that there is a relationship between the explanatory vector and the response variable.

This is the basic insight that underlies ANOVA. It is used to construct formal ANOVA tests to determine whether or not there is evidence of a relationship between the explanatory vector and the response variable. By conditioning on parts of the explanatory vector, this basic method can also be used to test for a relationship between particular subsets of explanatory variables and the response variable. In summary, ANOVA is a particular method that is used within the context of regression analysis to test for relationships between variables.

Answer 2

Yes, they are both OLS linear regression, but:

• $X_i$ in the first regression is a continuous predictor. You can subtitle whatever your values you have in your data set into the equation.
• $X_i$ in the second regression is a dummy variable. It's $0$ if the response belongs to group 1 otherwise it's $1$.

The look similar but they have different interpretation.

• $b$ in the first model is rate of change between the predictor and response, and are both assumed to be continuous.
• $b$ in the second model is the treatment effort of the second group relative to the first group. It is a regression for categorical predictor.

The models are not the same. The design matrix for the first and second model is different.