1
$\begingroup$

Let's say I have a huge dataset with 2000 observations. I want to test the significance of one predictor in a SLR Model. Then the F-statistic effectively becomes:

F = $\frac{\frac{\sum{(\hat{Y_i}-\bar{Y_i})^2}}{1}}{\frac{(Y_i-\hat{Y_i})^2}{1998}}$

or

F = $\frac{\frac{SSR}{1}}{\frac{SSE}{1998}}$ (aka MSR / MSE)

In this situation, the denominator of the denominator is WAY larger than the denominator of the numerator. This means that if your SLR model's R-squared was only 1%, your F-statistic would be $\frac{1998}{99}=20.182$ (Recall that $R^2 = \frac{SSR}{SSR + SSE}$). This is way larger than 1, and the p-value is approximately zero, indicating great significance.

Does this seem right? Our model has such a low R-squared, yet the p-value is microscopically low. It seems like this test would have huge a Type-I error rate in determining significance when the numerator's denominator (equal to number of predictors added to model) is far smaller than the denominator's denominator (equal to the degrees of freedom in error).

And if this is not erroneous, then how does one explain to a manger that a certain predictor is significant, despite such a low corresponding R-squared value?

Thanks a lot for your help.

$\endgroup$
  • 1
    $\begingroup$ As $n \to \infty$, any statistical test will become significant, not just $F$-tests. This is why a $p$-value alone isn't very compelling evidence and you should always include effect sizes (or in this case $R^2$). $\endgroup$ – Frans Rodenburg Jul 31 '18 at 1:38
  • $\begingroup$ Then what does one use to determine significance of regression coefficients, if not p-values? Just rules of thumb about what R-squared values are significant enough? $\endgroup$ – Greg Aug 6 '18 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.