What is the relationship between R-squared and p-value in a regression?

Question

tl;dr - for OLS regression, does a higher R-squared also imply a higher P-value? Specifically for a single explanatory variable (Y = a + bX + e) but would also be interested to know for n multiple explanatory variables (Y = a + b1X + ... bnX + e).

Context - I'm performing OLS regression on a range of variables and am trying to develop the best explanatory functional form by producing a table containing the R-squared values between the linear, logarithmic, etc., transformations of each explanatory (independent) variable and the response (dependent) variable. This looks a bit like:

Variable name --linear form-- --ln(variable) --exp(variable)-- ...etc

Variable 1 ------- R-squared ----R-squared ----R-squared --
...etc...

I'm wondering if R-squared is appropriate or if P-values would be better. Presumably there is some relationship, as a more significant relationship would imply higher explanatory power, but not sure if that is true in a rigorous way.

Answer 1

The answer is no, there is no such regular relationship between $R^2$ and the overall regression p-value, because $R^2$ depends as much on the variance of the independent variables as it does on the variance of the residuals (to which it is inversely proportional), and you are free to change the variance of the independent variables by arbitrary amounts.

As an example, consider any set of multivariate data $((x_{i1}, x_{i2}, \ldots, x_{ip}, y_i))$ with $i$ indexing the cases and suppose that the set of values of the first independent variable, $\{x_{i1}\}$, has a unique maximum $x^*$ separated from the second-highest value by a positive amount $\epsilon$. Apply a non-linear transformation of the first variable that sends all values less than $x^* - \epsilon/2$ to the range $[0,1]$ and sends $x^*$ itself to some large value $M \gg 1$. For any such $M$ this can be done by a suitable (scaled) Box-Cox transformation $x \to a((x-x_0)^\lambda - 1)/(\lambda-1))$, for instance, so we're not talking about anything strange or "pathological." Then, as $M$ grows arbitrarily large, $R^2$ approaches $1$ as closely as you please, regardless of how bad the fit is, because the variance of the residuals will be bounded while the variance of the first independent variable is asymptotically proportional to $M^2$.

---

You should instead be using goodness of fit tests (among other techniques) to select an appropriate model in your exploration: you ought to be concerned about the linearity of the fit and of the homoscedasticity of the residuals. And don't take any p-values from the resulting regression on trust: they will end up being almost meaningless after you have gone through this exercise, because their interpretation assumes the choice of expressing the independent variables did not depend on the values of the dependent variable at all, which is very much not the case here.

Answer 2

This answer doesn't directly deal with the central question; it's nothing more than some additional information that's too long for a comment.

I point this out because econometricstatsquestion will no doubt encounter this information, or something like it at some point (stating that $F$ and $R^2$ are related) and wonder if the information given in other answers here is wrong - it's not wrong - but I think it pays to be clear about what's going on.

There is a relationship under a particular set of circumstances; if you hold the number of observations and the number of predictors fixed for a given model, $F$ is in fact monotonic in $R^2$, since

$$ F = \frac{R^2/(k-1)}{(1-R^2)/(N-k)} $$

(If you divide numerator and denominator by $R^2$, and pull the constants in $k$ out, you can see that $1/F \propto 1/R^2 - 1$ if you hold $N$ and $k$ constant.)

Since for fixed d.f. $F$ and the p-value are monotonically related, $R^2$ and the $p$-value are also monotonically related.

But change almost anything about the model, and that relationship doesn't hold across the changed circumstances.

For example, adding a point makes $(N-k)/(k-1)$ larger and removing one makes it smaller but doing either can increase or decrease $R^2$, so it looks like $F$ and $R^2$ don't necessarily move together if you add or delete data. Adding a variable decreases $(N-k)/(k-1)$ but increases $R^2$ (and vice-versa), so again, $R^2$ is not necessarily related to $F$ when you do that.

Clearly, once you compare $R^2$ and $p$-values across models with different characteristics, this relationship doesn't necessarily hold, as whuber proved in the case of nonlinear transformations.

Answer 3

"for OLS regression, does a higher R-squared also imply a higher P-value? Specifically for a single explanatory variable (Y = a + bX + e) "

Specifically for a single explanatory variable, given the sample size, the answer is yes. As Glen_b has explained, there is a direct relationship between $R^2$ and the test statistic (be it a $F$ or $t$). For instance, as explained in this other question (High $R^2$ squared and high $p$-value for simple linear regression) for the simple linear regression with one covariate (and a constant), the relationship between $t$ and $R^2$ is:

$|t| = \sqrt{\frac{R^2}{(1- R^2)}(n -2)}$

So in this case, once you fix $n$, the higher the $R^2$ the higher the $t$ statistic and the lower the p-value.

"but would also be interested to know for n multiple explanatory variables (Y = a + b1X + ... bnX + e)."

The answer is the same, but instead of looking at one variable only, we now look at all variables together -- hence the $F$ statistic, as Glen_b has shown. And here you have to fix both $n$ and the number of parameters. Or, to put it better, fix the degrees of freedom.

Context - I'm performing OLS regression on a range of variables and am trying to develop the best explanatory functional form (...)

Ok, so this is actually a different problem. If you are looking at the best explanatory functional form, you should also take a look at cross-validation techniques. Even if $R^2$ is the quantity of interest for your problem (it usually isn't), finding the best fit in-sample can be very misleading -- you usually want your findings to generalize out of sample, and proper cross-validation can help you not overfit your data too much.

And here I'm guessing that you want "predictive" power (since you say you want to find "the best explanatory functional form"). If you want to do causal inference, for instance, then the $R^2$ or other predictive performance metrics are of little help without more structural/substantive knowledge of the problem.