Relationship between regression model prediction accuracy and p-value [duplicate]

Question

I'm hoping that somebody can help me understand the intuition in relation to p-values and model accuracy.

I have developed a linear regression model with multiple variables (continuous and categorical).

All my variables are highly significant with p-values equal to p-value = 2e-16 or similar.

My R-sqd, RMSE, MAE and MAPE are all quite low. R-sqd = 0.3 and MAPE = 0.7. RMSE is also quite high hence the model accuracy is quite low.

This feels counter-intuitive as a set of highly significant predictors should provide a relatively accurate model.

Can someone please help me understand if my intuition above is correct and the relationship between p-values and model accuracy?

Thanks

Answer 1

These two concepts are not necessarily related. A p-value gives you a measure of strength of association between your y and x, while predictive accuracy is in relation to how good the model predictions are.

Below is a simple example, y and x are highly related (on average), but the model is trash (for predicting), because of high variability in y, so the predictions will be quite bad (notice low $R^2$).

> x=seq(0,1000,1)
> y=rnorm(length(x),x,1000)
> summary(lm(y~x))

    Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -6.7322    61.3748  -0.110    0.913    
x             0.9593     0.1063   9.026   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 971.6 on 999 degrees of freedom
Multiple R-squared:  0.0754,    Adjusted R-squared:  0.07448 
F-statistic: 81.47 on 1 and 999 DF,  p-value: < 2.2e-16