I'm hoping that somebody can help me understand the intuition in relation to p-values and model accuracy.

I have developed a linear regression model with multiple variables (continuous and categorical).

All my variables are highly significant with p-values equal to p-value = 2e-16 or similar.

My R-sqd, RMSE, MAE and MAPE are all quite low. R-sqd = 0.3 and MAPE = 0.7. RMSE is also quite high hence the model accuracy is quite low.

This feels counter-intuitive as a set of highly significant predictors should provide a relatively accurate model.

Can someone please help me understand if my intuition above is correct and the relationship between p-values and model accuracy?



These two concepts are not necessarily related. A p-value gives you a measure of strength of association between your y and x, while predictive accuracy is in relation to how good the model predictions are.

Below is a simple example, y and x are highly related (on average), but the model is trash (for predicting), because of high variability in y, so the predictions will be quite bad (notice low $R^2$).

> x=seq(0,1000,1)
> y=rnorm(length(x),x,1000)
> summary(lm(y~x))

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -6.7322    61.3748  -0.110    0.913    
x             0.9593     0.1063   9.026   <2e-16 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 971.6 on 999 degrees of freedom
Multiple R-squared:  0.0754,    Adjusted R-squared:  0.07448 
F-statistic: 81.47 on 1 and 999 DF,  p-value: < 2.2e-16
  • $\begingroup$ Wow! Thank you very much for the quick response and taking time to help me understand this with an example. I really appreciate your help. $\endgroup$ Feb 26 '20 at 9:56
  • 1
    $\begingroup$ I believe few people would characterize a p-value as "a measure of strength of association." One reason is that the p-value depends on the sample size whereas the strength of association is a population property. $\endgroup$
    – whuber
    Aug 2 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.