# Relationship between regression model prediction accuracy and p-value

I'm hoping that somebody can help me understand the intuition in relation to p-values and model accuracy.

I have developed a linear regression model with multiple variables (continuous and categorical).

All my variables are highly significant with p-values equal to p-value = 2e-16 or similar.

My R-sqd, RMSE, MAE and MAPE are all quite low. R-sqd = 0.3 and MAPE = 0.7. RMSE is also quite high hence the model accuracy is quite low.

This feels counter-intuitive as a set of highly significant predictors should provide a relatively accurate model.

Can someone please help me understand if my intuition above is correct and the relationship between p-values and model accuracy?

Thanks

• Apr 2 at 13:17

These two concepts are not necessarily related. A p-value gives you a measure of strength of association between your y and x, while predictive accuracy is in relation to how good the model predictions are.

Below is a simple example, y and x are highly related (on average), but the model is trash (for predicting), because of high variability in y, so the predictions will be quite bad (notice low $$R^2$$).

> x=seq(0,1000,1)
> y=rnorm(length(x),x,1000)
> summary(lm(y~x))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  -6.7322    61.3748  -0.110    0.913
x             0.9593     0.1063   9.026   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 971.6 on 999 degrees of freedom
Multiple R-squared:  0.0754,    Adjusted R-squared:  0.07448
F-statistic: 81.47 on 1 and 999 DF,  p-value: < 2.2e-16

• Wow! Thank you very much for the quick response and taking time to help me understand this with an example. I really appreciate your help. Feb 26 '20 at 9:56
• I believe few people would characterize a p-value as "a measure of strength of association." One reason is that the p-value depends on the sample size whereas the strength of association is a population property.
– whuber
Aug 2 at 1:18