Does the overall p-value of a multiple regression trump the individual ones?

Question

Here's my situation:

In theory, $y = f(x1, x2,...,x5)$, and the exact equation is context dependent.

In practice, I don't have data for $x4$ and $x5$ as yet.

Based on the somewhat promising results of analysis with $y$, $x1$, $x2$, and $x3$, I'd like to make the case for further investment to obtain data for $x4$ and $x5$ and repeat the analysis. If that provides a better fit, it could lead to actionable insights.

Regression results so far on a dataset with 100+ samples look like this:

y ~ x1 has $R^2$ of 42% with p-value less than 5%

y ~ x1+x2 has $adj.R^2$ of 48% with p-value of x1 and overall p-value less than 5% but p-value of x2 is 20%

y ~ x1+x3 has $adj.R^2$ of 45% with p-value of x1 and overall p-value less than 5% but p-value of x3 is 13%

y ~ x1+x2+x3 has $adj.R^2$ of 51% with p-value of x1 and overall p-value less than 5% but p-value of x2 is 20% and x3 is 13%

Residual plots for all of the above show significant bias which I attribute to missing $x4$ and $x5$.

Given that the overall p-values are always less than 5%, is it okay to conclude that the analysis so far explains 51% of the variation in $y$ or should I be mindful of the not-so-great p-values of $x2$ and $x3$ and go with 42%?

Answer 1

Answer from my comments

In looking at a model, you can look at the p-value for the overall model without being overly concerned about the p-values for the individual terms.

If you have reason to think a model should include x1, x2, and x3, or include x1, x2, x3, x4, and x5, then you have reason to fit the model with the variables you think should be included.

An increase in adjusted r-squared may be informative, especially if you don't have any clear reason to include or not include certain variables. You might also use AIC, BIC, or AICc to evaluate models.

Also note that p = 0.05 isn't a magic cutoff. In the case of multiple regression, you might decide that a p-value of, say, < 0.20 is potentially "interesting", if you want to use p values as some sort of guide.

The "bias" you see in the residuals may suggest that you should include x4 and x5, or it may suggest that your model isn't the best for other reasons. Perhaps you can add polynomials, transformations, or interactions of your variables. Or perhaps a generalized linear model may be more appropriate for your situation.