I was playing around with a simple linear models when I noticed that, in the ANOVA table, the ratio MSreg/MSres does not exactly correspond to the F-value. Indeed, the two values are very similar but not the same. Here my script

#quick view of the dataset
> head(my_data)
  Diameter Height
1    0.325  0.080
2    0.320  0.100
3    0.280  0.110
4    0.125  0.040
5    0.400  0.135
6    0.335  0.100

#setting up the lm()
> ls1 <- lm(Diameter~Height, data=my_data)
> anova(ls1)
Analysis of Variance Table

Response: Diameter
          Df  Sum Sq Mean Sq F value    Pr(>F)    
Height     1 0.82415 0.82415  602.63 < 2.2e-16 ***
Residuals 98 0.13402 0.00137                      
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here 0.82415/0.00137=601.5693 which is not the F value in the table. Is there a particular reason for that?

  • $\begingroup$ It's an issue with loss of numerical precision due to rounding. Try (0.82415 / 1 ) / (0.13402 / 98). $\endgroup$
    – dipetkov
    Apr 25, 2022 at 12:57

1 Answer 1


The number 0.00137 is rounded (hopefully, rather than truncated) to around three figures of accuracy (actually, if it's rounded, the % error could be nearly half a percent -- the starting digit is only "1", so it's really somewhat less than three figures, perhaps nearer to two).

The accuracy of your calculated ratio using that number should not be expected to have greater accuracy, but it could have less (imagine the numerator rounded up while the denominator rounded down, for example). So to three figures (at best) you have 602 give or take something in the ballpark of 1 or so.

i.e. everything is about what it should be to the accuracy we can expect.

You did the calculation at the end to 7 figures. If you want close to 7 figure accuracy you'll need at least 7 figures in the numbers you calculated it from (as long as you didn't do any calculation that would lead to a larger loss of significant figures, like subtracting two almost-equal numbers). For a simple ratio like this aim for one or two more figures than you want to have at the end. For something a bit less stable you may need more again.

In that case, the rounded numbers in the table are not sufficient for the purpose calculation check you want to do. If you want to check the calculations to say 4 or 5 digits, you would need to extract more figures for the mean squares that are the inputs to that ratio (this is easy to do in R).




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.