Common statistical tests as linear models

(UPDATE: I dived deeper into this and and posted the results here)

The list of named statistical tests is huge. Many of the common tests rely on inference from simple linear models, e.g. a one-sample t-test is just y = β + ε which is tested against the null model y = μ + ε i.e. that β = μ where μ is some null value - typically μ=0.

I find this to be quite a bit more instructive for teaching purposes than rote learning named models, when to use them, and their assumptions as if they had nothing to do with each other. That approach promotes does not promote understanding. However, I cannot find a good resource collecting this. I am more interested in equivalences between the underlying models rather than the method of inference from them. Although, as far as I can see, likelihood ratio tests on all these linear models yield the same results as the "classical" inference.

Here are the equivalences I've learned about so far, ignoring the error term $$\varepsilon \sim \mathcal N(0, \sigma^2)$$ and assuming that all null hypotheses are the absense of an effect:

One-sample t-test: $$y = \beta_0 \qquad \mathcal{H}_0: \beta_0 = 0$$.

Paired-sample t-test: $$y_2-y_1 = \beta_0 \qquad \mathcal{H}_0: \beta_0 = 0$$

This is identical to a one-sample t-test on pairwise differences.

Two-sample t-test: $$y = \beta_1 * x_i + \beta_0 \qquad \mathcal{H}_0: \beta_1 = 0$$

where x is an indicator (0 or 1).

Pearson correlation: $$y = \beta_1 * x + \beta_0 \qquad \mathcal{H}_0: \beta_1 = 0$$

Notice the similarity to a two-sample t-test which is just regression on a binary x-axis.

Spearman correlation: $$rank(y) = \beta_1 * rank(x) + \beta_0 \qquad \mathcal{H}_0: \beta_1 = 0$$

This is identical to a Pearson correlation on rank-transformed x and y.

One-way ANOVA: $$y = \beta_1*x_1 + \beta_2*x_2 + \beta_3*x_3 +... \qquad \mathcal{H}_0: \beta_1, \beta_2, \beta_3, ... = \beta$$

where $$x_i$$ are indicators selecting the relevant $$\beta$$ (one $$x$$ is 1; the others are 0). The model could probably be written in matrix form as as $$Y = \beta * X$$.

Two-way ANOVA: $$y = \beta_1 * X_1 + \beta_2 * X_2 + \beta_3 * X_1 * X_2 \qquad \mathcal{H}_0: \beta_3 = 0$$

for two two-level factors. Here $$\beta_i$$ are vectors of betas where one is selected by the indicator vector $$X_i$$. The $$\mathcal{H}_0$$ shown here is the interaction effect.

Could we add more "named tests" to this list of linear models? E.g., multivariate regression, other "non-parametric" tests, binomial tests, or RM-ANOVAs?

UPDATE: questions have been asked and answered about ANOVA and t-tests as linear models here on SO. See this question and tagged related questions.

• I think these comparisons are appropriate but that at some point there are also subtle differences. E.g. take the one-way ANOVA: where a linear regression will provide you with the coefficients and in most software packages the significance per coefficient with Wald tests (which might not be appropriate), an ANOVA will provide a single p-value indicating whether any one of the coefficients is significantly different from zero. A Likelihood Ratio Test between a null model and the regression model of interest might be more comparable. As such, I wouldn't completely equalize these tests/models.
– IWS
Sep 15, 2017 at 8:04
• Good point; I updated the question, saying that "I am more interested in equivalences between the underlying models rather than the method of inference from them." Likelihood-ratio tests on the one-way ANOVAs and interaction terms yields identical p-values as the "classical" analyses as far as my testing goes. Sep 15, 2017 at 8:20
• Fair enough, but inference aside, do note that regression models also provide added flexibility when handling non-linearity (although transformations might be also tested with these 'named tests', splines are a different matter) or handling heteroscedasticity, not even mentioning the family of generalized models which also handle non-continuous dependent variables. Nonetheless, I can see explaining the named tests as restrictive variations of regression models for teaching purposes can have merit, so +1
– IWS
Sep 15, 2017 at 8:26
• Is Spearman rank correlation really a linear model ? Nov 29, 2017 at 9:22
• @MartinDietz: Yes, after rank-transforming x and y, it is linear. R code: x = rnorm(100); y = rnorm(100); summary(lm(rank(x) ~ rank(y))); cor.test(x, y, method='spearman') Nov 29, 2017 at 9:31

The Cochran-Armitage test of trend can be formulated by: $$E[\mbox{logit} (p) | t] = \beta_0 + \beta_1 t \qquad \mathcal{H}_0: \beta_1 = 0$$
The Pearson Chi-Square test of independence for a $p \times k$ contingency table is a log-linear model for the cell frequencies given by:
$$E[\log (\mu)] = \beta_0 + \beta_{i.} + \beta_{.j} + \gamma_{ij} \quad i,j > 1 \qquad\mathcal{H}_0: \gamma_{ij} = 0, \quad i,j > 1$$