This is the data set:


I'm looking at a simple linear regression. The data is quite trivial.

import pandas as pd
from statsmodels.formula.api import ols

nt = pd.read_csv('normtemp.csv')
model = ols('hr ~ temp', data=nt).fit()


   temp  hr
0  96.3  70
1  96.7  71
2  96.9  74
3  97.0  80
4  97.1  73

To test whether there is lack of fit, I've seen the F-test done this way:

from statsmodels.stats.api import anova_lm

nt_fact = nt.copy()
nt_fact['temp'] = nt_fact['temp'].astype(object)
model_fact = ols('hr ~ temp', data=nt_fact).fit()

an_out = anova_lm(model, model_fact)


   df_resid          ssr  df_diff      ss_diff         F    Pr(>F)
0     126.0  6009.601776      0.0          NaN       NaN       NaN
1      96.0  4177.428932     30.0  1832.172843  1.403484  0.110304

Same thing in R:

linear.model <- lm( hr ~ temp)
full.model <- lm(hr ~ factor(temp))
anova(linear.model, full.model)

The p-value from ANOVA is 0.1103, do not reject H0, there is not significant evidence that the linear model is not appropriate.

EDIT: This is not about overfitting. This is to check whether the linear model is or is not a good fit for the data. Rejecting H0 from ANOVA would indicate the linear model is not a good fit for the data.

I am blanking completely on the intuitive explanation for why this works. We build two models here - the actual LR model where the predictor is continuous, and another model where the predictor is converted to categorical (and that's the only change) but the data is otherwise the same.

What exactly are we comparing here with ANOVA, and why rejecting H0 would indicate the fit is not good? (or the other way around)

  • 1
    $\begingroup$ @cdalitz a linear model is effectively nested inside a model that treats the variable as a factor variable. Alternatively you could use a polynomial with an equal amount of parameters as categories. $\endgroup$ Jun 28, 2022 at 5:28
  • 1
    $\begingroup$ Of possible interest $\endgroup$
    – Dave
    Jun 28, 2022 at 10:03
  • 1
    $\begingroup$ @cdalitz I disagree that the models are not nested. With the factor model you can describe any linear model that you like. The solution space of the linear model is inside the solution space of the factor model. Otherwise, if you don't believe it, then try to add the variable temp to the regressor matrix factor(temp) and find out whether the rank of the regressor matrix remains the same or not. $\endgroup$ Jun 29, 2022 at 14:24
  • 1
    $\begingroup$ You do not need to have the exact same variables to have a nested model. For example the following models are nested $$y = a + bx + \epsilon$$ and $$y = c (x-1)^2 + d(x-1) + ex^2$$ the first model is a 1st order polynomial, and the second is a 2nd order polynomial. Any solution of the first can be described with the second, even though the second does not have exactly the same variables. $\endgroup$ Jun 29, 2022 at 14:30
  • 1
    $\begingroup$ @cdalitz nesting does not mean that one set of covariates are subsets of the other. Rather it means the linear space spanned by a design matrix is a subset (subspace) of linear spaces spanned by the other design matrix. Yeah, that confused me too, until I looked how the ANOVA test was formed more rigorously. $\endgroup$
    – s l
    Jul 1, 2022 at 4:22

3 Answers 3


TLDR; Bascially this anova to test the goodness of fit is effectively testing whether the conditional means of the residuals are different from zero. A good model is supposed to model the conditional mean of the data and reduce the expectation value of the conditional mean of the residuals to zero.

I am blanking completely on the intuitive explanation for why this works. We build two models here - the actual LR model where the predictor is continuous, and another model where the predictor is converted to categorical (and that's the only change) but the data is otherwise the same.

What exactly are we comparing here with ANOVA, and why rejecting H0 would indicate the fit is not good? (or the other way around)

The image below might give an intuitive view what this model is testing.


Scatter plot hr vs temp

In the first plot we see the scatter plot along with the two models.

  • The linear model (the green line).

    Is the linear model (the green line) a good model?

    This is not a question about whether the linear model explains ever single data point. (It doesn't explain a lot and the coefficient of determination is small $R^2 =0.06$).

    Instead, it is about whether the linear model explains the expectation value of the heart rate as a function of the temperature.

  • The full model (red line).

    So for the goodness of fit it is not about the individual points (that is what the coefficient of determination is expressing), but instead it is about the expectation value of the points.

    The full model, the red line, is the estimate of this expectation value evaluated separately for each single temperature point.

Now, the question is whether the full model (the red line) is doing 'better' than the linear model (the green line). This 'better' might mean: does it have a lower variance of the residuals. If the full model is doing better, then this means that the linear model is not a model that completely describes the expectation value (however, this doesn't mean it is a bad model, it means that it might not be perfect).

Since the full model (red line) is more flexible it will fit the random noise as well and it will always do somewhat better even when the linear model (green line) is the true model. So more precisely is the question whether the full model is doing statistical significantly better. Is the full model (red line) doing better beyond what might be expected in comparison to the null hypothesis, which is that the linear model (green line) is the true model?

To test this statistical significance we use the variance of the noise. We see that even with the full model (red line) there is variation in the data points. If the null model is true, then the variation of the noise should explain the variation in the difference between the red line and the green line (think for instance about how you estimate the variance of the mean based on the residuals).

the question becomes, is the variation of the means larger or not than what we expect based on the variation of the residuals. So that is your ANOVA/F-test in a nutshell, a comparison of the variance in the residuals and the variance in the means of the residuals.

So in a certain sense this F-test (which compares variances) is a test for whether the expectation value of the residuals is different from zero. This we see in the second plot.

Residual plot

In the second plot we see a scatter plot for the residuals. It is the difference of the observations with the estimation based on the linear model.

Also in this plot, is the difference between the linear model and the full model. What we see here is that there is not much of a clear trend (this is another way to test goodness of fit: test the residual plot visually).

We can also test this with an ANOVA but now using a model for the residuals. In R this will use the following:

 linear.model <- lm( hr ~ temp)
 full.model <- lm(hr ~ factor(temp))

 res = residuals(linear.model)  

 modf <- lm(res ~ factor(temp)) 
 mod0 <- lm(res ~ 1)

 anova(linear.model, full.model)

which gives the same tables

 > anova(linear.model, full.model)
 Analysis of Variance Table

 Model 1: hr ~ temp
 Model 2: hr ~ factor(temp)
   Res.Df    RSS Df Sum of Sq      F Pr(>F)
 1    126 6009.6                           
 2     96 4177.4 30    1832.2 1.4035 0.1103

 > anova(mod0,modf)
 Analysis of Variance Table

 Model 1: res ~ 1
 Model 2: res ~ factor(temp)
   Res.Df    RSS Df Sum of Sq      F Pr(>F)
 1    127 6009.6                           
 2     96 4177.4 31    1832.2 1.3582 0.1315

There is only a difference in degrees of freedom. The reason for this is because the residuals are, if the null hypothesis is correct, not the same as independent Gaussian noise. The residuals are in a space with one degree of freedom less (see Why are the residuals in $\mathbb{R}^{n-p}$?).

  • $\begingroup$ This is the explanation I was looking for. It's very clear now. Thank you. $\endgroup$ Jul 1, 2022 at 2:16

Here is the explanation of how the ANOVA works, based of off Bickel and Doksum's Mathematical Statistics (1977), Chapter 7. This is a geometric view (read as Linear Algebraic view) and hopefully can provide some intuition on what exactly the F-test and ANOVA is doing. Some prerequisite understanding required is the concept of linear subspaces, image spaces, and projections, orthogonality of subspaces, and norms of vectors.

Continuous Model

Your continuous model has the form

$Y_i = \beta_0 + \beta_1 X_{1,i} + \epsilon_i$

where $X_{1,i}$ is the i-th observation of temperature. Your design matrix looks like: $X_{cont} = \begin{bmatrix} 1 & x_{1,1} \\ 1 & x_{1,2} \\ \vdots & \vdots \\ 1 & x_{1,n}\end{bmatrix}$ where your $x_{1,i}$ for $i = 1,...,n$ are your temperature observations.

Categorical Case

In the categorical case, your design matrix will look different. You model will now be of the form: $Y_i = \beta_0 + \beta_1T_{2,i} + \beta_2 T_{3,i} + ... + \beta_{n-1} T_{n,i} + \epsilon_i$. Note that $T_{1,i}$ is not in the model, as it has been pooled into the $\beta_0$ term. We cannot have $\beta_0$ exist along with coefficient for all levels of your categorical temperature for model identifiable issues assuming OLS where $\epsilon_i \sim N(0,\sigma^2)$.

So your design matrix will look the following, in the case of your data set with 5 distinct categories : $X_{cat} = \begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 & 0 \\ 1 & 0 & 1 & 0 & 0 \\ 1 & 0 & 0 & 1 & 0 \\ 1 & 0 & 0 & 0 & 1\end{bmatrix}$ and so on. Of course this means your estimate of $E(Y|T=T_1)$ is $\beta_0$ and your estimate for $E(Y|T=T_i)$ where $i > 1$ is $\beta_0 + \beta_{i-1}$.

Connecting the two cases

Note that in your case with 5 categories, $X_{cont} = \begin{bmatrix} 1 & x_{1,1} \\ 1 & x_{1,2} \\ 1 & x_{1,3} \\ 1 & x_{1,4} \\ 1 & x_{1,5}\end{bmatrix}$, where $x_{1,i}$ is assumed to be any real number (of course, this is an assumption that may not always hold depending how extreme your data is, but in general, is the assumption for OLS). The column span (image space) of $X_{cont}$ lies in the image space of $X_{cat}$. To see this: consider the the image space of $X_{cat}$ and $X_{cont}$.

For the image space of the design matrix in the categorical case:

$\omega := Img(X_{cat}) = \lbrace{\mathbf{v} \in R^5 : \alpha_1 \mathbf{1} + \alpha_2 \mathbf{e}_2 + \alpha_3 \mathbf{e}_3 + \alpha_4 \mathbf{e}_4 + \alpha_5 \mathbf{e}_5\rbrace}$, where $\mathbf{1}$ is a 1 x 5 vector of ones, and $e_i$ is the i-th standard basis vector; and $\alpha_j$ can take on any real number. In short, these are just the columns of $X_{cat}$ put into vector form, and we are just taking linear combinations of these vectors to get the image space.

For the image space of the design matrix in the continuous case,

$\omega_0 := Img(X_{cont}) = \lbrace{\gamma_1 \mathbf{1} + \gamma_2 \mathbf{x} \rbrace}$ where $\mathbf{x}$ is the 1 x 5 column vector of temperature observations and $\mathbf{x}$ looks like this $\begin{bmatrix}x_{1,1} \\ x_{1,2} \\ x_{1,3}\\ x_{1,4} \\ x_{1,5}\end{bmatrix}$. Again, this is just taking the columns of $X_{cont}$ as vectors, and taking linear combinations of such vectors.

If I look at image space of $X_{cat}$, and I set $\alpha_1 = x_{1,1}$ and $\alpha_2 = x_{1,2}-x_{1,1}$ and so on, so that we end with $\alpha_5 = x_{1,5}-x_{1,1}$, then we get a vector $\begin{bmatrix}x_{1,1} \\ x_{1,2} \\ x_{1,3}\\ x_{1,4} \\ x_{1,5}\end{bmatrix}$. So clearly this vector lies in the image space of $X_{cat}$.

Now if I look at the image space of $X_{cont}$, and I set $\gamma_1 = 0$ and $\gamma_1 = 1$, then I also get the same vector $\begin{bmatrix}x_{1,1} \\ x_{1,2} \\ x_{1,3}\\ x_{1,4} \\ x_{1,5}\end{bmatrix}$. In fact, any vector in the image space of $X_{cont}$ is in the image space of $X_{cat}$. Thus we say the continuous model is "nested" in the categorical model, in the sense that the $\omega$ and $\omega_0$ have a nested relationship.

Technical clarification: (in order to call an image space a "subspace", we must allow the image space to contain the 0, and be closed under vector addition and scalar multiplication. Thus we must place conditions on $\alpha_i$ and $\gamma_i$ to meet these requirements of a subspace. Here is not an issue as we can simply define the $\alpha_i$ and $\gamma_i$ as elements of R).


ANOVA is defined by Bickel and Docksum in terms of projections. I do not have the book on hand right now, but I will update this answer to provide page numbers.

The ANOVA produces a test statistic given by $\frac{||P_\omega Y - P_{\omega_0}Y||^2/(\dim \omega - \dim \omega_0)}{||Y-P_{\omega} Y||^2/(n-\dim\omega)}$. Here $P_{\omega}$ is the projection of your $Y$ responses onto the $\omega$ linear space, and likewise for $P_{\omega_0}$. But what is happening here?

First note $P_{\omega}Y$ essentially represents your regression estimates in the categorical model. Then $P_{\omega_0}Y$ represents your regression estimates in the continuous model. Also note that $Y-P_{\omega_0}Y$ is the vector of residual of the continuous model and $Y-P_{\omega}Y$ is the vector of residuals in the categorical model. Of course we can square them to get sum of square errors. Now we claim that $||Y-P_{\omega_0}Y||^2 = ||Y-P_{\omega}Y||^2 + ||P_\omega Y - P_{\omega_0}Y||^2$. You can easily show this by noting that $Y-P_\omega Y$ is orthogonal to any vector in $\omega$ and that since $P_\omega Y - P_{\omega_0}Y$ must also lie in $\omega$, it is also orthogonal to $Y-P_\omega Y$, so it goes like this: \begin{align*} Y-P_{\omega_0}Y &= Y- P_{\omega}Y + P_{\omega}Y - P_{\omega_0}Y \\ ||Y-P_{\omega_0}Y||^2 &= ||Y- P_{\omega}Y + P_{\omega}Y - P_{\omega_0}Y||^2 \\ &= ||Y- P_{\omega}Y||^2 + ||P_{\omega}Y - P_{\omega_0}Y||^2 && \text{Pythagorean Theorem} \end{align*} So some algebra will show us that $||P_{\omega}Y - P_{\omega_0}Y||^2 = ||Y-P_{\omega_0}Y||^2 - ||Y- P_{\omega}Y||^2$ which is the difference in residuals between the two models. This makes sense since $||Y-P_{\omega_0}||^2$ is expected to be larger than a more refined model with SSE given by $||Y-P_{\omega}||^2$.

So now we know that the ANOVA where $F = \frac{||P_\omega Y - P_{\omega_0}Y||^2/(\dim \omega - \dim \omega_0)}{||Y-P_{\omega} Y||^2/(n-\dim\omega)}$ is really $\frac{(||Y-P_{\omega_0}Y||^2 - ||Y- P_{\omega}Y||^2)/(\dim \omega - \dim \omega_0)}{||Y-P_{\omega} Y||^2/(n-\dim\omega)}$. (We divide by some factors to deal with MSE rather than SSE as a way to standardize and also as a way to make the statistical distribution into an F distribution, for reasons that should be for another topic.)

Conclusion and TLDR

So now we see that the ANOVA is really just comparing how much better a more refined model (given by subspace $\omega$) will perform, in terms of residuals, when compared to a less refined model (given by subspace $\omega_0$). And "model 2 is more refined than model 1" is the same as saying the image space of model 2 contains the image space of model 1, or $\omega_0 \subseteq \omega$. The nesting requirement is critical so that we may invoke orthogonality and pythagorean theorem when decomposing $||Y-P_{\omega_0}Y||^2$.

In the context of your data set

Your output shows a p-value of 0.11. This fails to reject the null hypothesis at the 0.05 level. Well what is the null?

Note that in the above exposition, we were essentially testing if your regression model, which estimates $E(Y|X)$, lies in the $\omega_0$ subspace or in the $\omega$ subspace, so we are really testing if $H_0: E(Y|X) \in \omega_0$ versus the alternate $H_a: E(Y|X) \in \omega$. Since we fail to reject $H_a$, we fall back to our null, which is $E(Y|X) \in \omega_0$, that is, the categorical model does not provide a significant reduction in SSE to warrant its use over the continuous model.

Rejecting the null means that the difference between the SSE's the is, the difference between $||Y-P_{\omega_0}Y||^2$ and $||Y-P_{\omega}Y||^2$ is large enough for one to be considered "better".

  • $\begingroup$ Very detailed explanation. Makes sense. Thank you. $\endgroup$ Jul 1, 2022 at 2:14
  • 1
    $\begingroup$ @FlorinAndrei This crux of the F test equation is equation 7.3.12 on page 276 (Bickel and Doksum Mathematical Statistics, 1977). The derivations are in the immediate pages before it. They use $\hat{\xi}$ to denote what I wrote as $P_{\omega} Y$ and $\hat{\xi}_0$ for my $P_{\omega_0} Y$. $\endgroup$
    – s l
    Jul 1, 2022 at 3:03
  • $\begingroup$ Semi-related: how would you describe the Bickel and Doksum book? For whom is it a must-have? $\endgroup$ Jul 1, 2022 at 17:08
  • 1
    $\begingroup$ @FlorinAndrei it is for those who wish to study mathematical statistics at a deeper level of at least an advanced masters student. The authors have recently published a newer version of the book, which does not contain this exposition on linear models in the 1977 version. Both are good, but the newer one skips on some traditional topics for some more "modern" topics. $\endgroup$
    – s l
    Jul 1, 2022 at 18:16

ANOVA is not a method for testing whether a model fit is ok, but for comparing different nested models. It is usually used to identify statistically significant predictors, but there are cases when this fails, most notably in the presence of (multi-)collinearity. "Nested models" are models for which one predictor set is a subset of the predictors of the other model. This is not the case in your comparison between your linear.model and full.model, which do not even have a single common predictor.

As a first step in evaluating your model, you should look at $R^2$ and the overall F-statistic. With all the data form your given reference the result is:

> fit <- lm(hr ~ temp, data=x)
> summary(fit)
             Estimate Std. Error t value Pr(>|t|)   
(Intercept) -179.1193    87.8417  -2.039   0.0435 * 
temp           2.5742     0.8944   2.878   0.0047 **

Multiple R-squared:  0.06169,   Adjusted R-squared:  0.05424 
F-statistic: 8.284 on 1 and 126 DF,  p-value: 0.004699

$R^2$ is suspiciously low and the model does a lousy predictive job, which is confirmed by plotting the data and the fitted model line:

enter image description here

The coefficient of temp is statistically significant (p-value of F-statistic is 0.0047, as is the overall F-statistic), however, so you can at least conclude that hs increases with temp, albeit wildly scattering around the predicted values.

The latter point is also confirmed by the very small $R^2$, which says that the random fluctuations in your model greatly dominate the systematic effects. The random fluctuations, on the other hand, seem to be consistent with a linear model, i.e., seem to be normal, which you can visualize with a QQ-Plot.

  • $\begingroup$ So the ANOVA trick is more like an additional test. It compares the actual linear regression with what is essentially a perfect-fit model, and concludes - yeah, they don't differ too much, therefore regression is not too far off. And it's done because R-squared is small, but residuals are normal and homoskedastic, so ANOVA is more like an extra assurance that we are getting some regression done. Where I've seen it, it was not a core test of the linear regression, it was more like this one extra thing that you could try on the side. $\endgroup$ Jun 21, 2022 at 17:24
  • $\begingroup$ No, ANOVA compares a model with a model that has one or more variables dropped (my description "subset of predictors" apparently was unnecessarily complicated). As your model has only one predictor, the only possible ANOVA test is omitting this predictor, but this is not an additionals test for a continuous predictor because the p-value is identical to the p-value of a t-test for a zero predictor coeffitient (0.0047 in this case). $\endgroup$
    – cdalitz
    Jun 21, 2022 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.