When doing statistics with linear regression, p-values and statistical significance are fundamental aspects for building a linear regression model.

But unlike in statistics, machine learning is all about making correct predictions. The more variables we add to a linear regression model, the higher we expect the machine learning accuracy to be.

Would you agree that p values and an analysis of statistical significance are not important during machine learning model evaluation when building a linear regression?

I ask because my professor during my masters degree made that point but I have seen data scientists spend a lot of time discussing p values.


I'm afraid, you are wrong. Suppose $$ Y = \alpha + \beta_1 X_1 + \beta_2 X_2 + \varepsilon, $$ $$ \varepsilon\sim N(0,\sigma^2). $$ Suppose further that $Z_1$, ..., $Z_k$ are "nuisance" candidate predictors, uncorrelated with $Y$. Gauss-Markov theorem proves that the best estimate of the model will be the regular OLS (ordinary least squares) estimate of coefficients $(\alpha, \beta_1, \beta_1)$ based on predictors $X_1$ and $X_2$. Any machine learning algorithm will achieve this result at best.

If the sample size is substantial, examination of p-values will be instrumental in identification of informative predictors $X_1$ and $X_2$ in the candidate set $\{X_1, X_2, Z_1, ..., Z_k\}$ and then following the Gauss-Markov approach.

Your machine learning algorithm may "care" all it wants about predictions but its predictions will be less accurate than those coming out of OLS linear regression because its model will be worse, generally speaking.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.