17
$\begingroup$

tl;dr - for OLS regression, does a higher R-squared also imply a higher P-value? Specifically for a single explanatory variable (Y = a + bX + e) but would also be interested to know for n multiple explanatory variables (Y = a + b1X + ... bnX + e).

Context - I'm performing OLS regression on a range of variables and am trying to develop the best explanatory functional form by producing a table containing the R-squared values between the linear, logarithmic, etc., transformations of each explanatory (independent) variable and the response (dependent) variable. This looks a bit like:

Variable name --linear form-- --ln(variable) --exp(variable)-- ...etc

Variable 1 ------- R-squared ----R-squared ----R-squared --
...etc...

I'm wondering if R-squared is appropriate or if P-values would be better. Presumably there is some relationship, as a more significant relationship would imply higher explanatory power, but not sure if that is true in a rigorous way.

$\endgroup$
15
$\begingroup$

The answer is no, there is no such regular relationship between $R^2$ and the overall regression p-value, because $R^2$ depends as much on the variance of the independent variables as it does on the variance of the residuals (to which it is inversely proportional), and you are free to change the variance of the independent variables by arbitrary amounts.

As an example, consider any set of multivariate data $((x_{i1}, x_{i2}, \ldots, x_{ip}, y_i))$ with $i$ indexing the cases and suppose that the set of values of the first independent variable, $\{x_{i1}\}$, has a unique maximum $x^*$ separated from the second-highest value by a positive amount $\epsilon$. Apply a non-linear transformation of the first variable that sends all values less than $x^* - \epsilon/2$ to the range $[0,1]$ and sends $x^*$ itself to some large value $M \gg 1$. For any such $M$ this can be done by a suitable (scaled) Box-Cox transformation $x \to a((x-x_0)^\lambda - 1)/(\lambda-1))$, for instance, so we're not talking about anything strange or "pathological." Then, as $M$ grows arbitrarily large, $R^2$ approaches $1$ as closely as you please, regardless of how bad the fit is, because the variance of the residuals will be bounded while the variance of the first independent variable is asymptotically proportional to $M^2$.


You should instead be using goodness of fit tests (among other techniques) to select an appropriate model in your exploration: you ought to be concerned about the linearity of the fit and of the homoscedasticity of the residuals. And don't take any p-values from the resulting regression on trust: they will end up being almost meaningless after you have gone through this exercise, because their interpretation assumes the choice of expressing the independent variables did not depend on the values of the dependent variable at all, which is very much not the case here.

$\endgroup$
10
$\begingroup$

This answer doesn't directly deal with the central question; it's nothing more than some additional information that's too long for a comment.

I point this out because econometricstatsquestion will no doubt encounter this information, or something like it at some point (stating that $F$ and $R^2$ are related) and wonder if the information given in other answers here is wrong - it's not wrong - but I think it pays to be clear about what's going on.

There is a relationship under a particular set of circumstances; if you hold the number of observations and the number of predictors fixed for a given model, $F$ is in fact monotonic in $R^2$, since

$$ F = \frac{R^2/(k-1)}{(1-R^2)/(N-k)} $$

(If you divide numerator and denominator by $R^2$, and pull the constants in $k$ out, you can see that $1/F \propto 1/R^2 - 1$ if you hold $N$ and $k$ constant.)

Since for fixed d.f. $F$ and the p-value are monotonically related, $R^2$ and the $p$-value are also monotonically related.

But change almost anything about the model, and that relationship doesn't hold across the changed circumstances.

For example, adding a point makes $(N-k)/(k-1)$ larger and removing one makes it smaller but doing either can increase or decrease $R^2$, so it looks like $F$ and $R^2$ don't necessarily move together if you add or delete data. Adding a variable decreases $(N-k)/(k-1)$ but increases $R^2$ (and vice-versa), so again, $R^2$ is not necessarily related to $F$ when you do that.

Clearly, once you compare $R^2$ and $p$-values across models with different characteristics, this relationship doesn't necessarily hold, as whuber proved in the case of nonlinear transformations.

$\endgroup$
  • $\begingroup$ I don't disagree with you, but it looks like you're answering a different question than I did. It took some reading, but I concluded that the question asks about what relationship, if any, holds between $p$ and $R^2$ when (caeteris paribus) independent variables are nonlinearly transformed. It is only when those variables are left unchanged--or, at most, linearly transformed among themselves--that we can say anything at all about such a relationship. That is part of the sense in which I think your qualifier "for a given model" has to be understood. $\endgroup$ – whuber Feb 21 '13 at 8:47
  • $\begingroup$ I am answering a different question; and I believe your interpretation of the meaning is correct. I was more worried that such an issue as the one I raised would lead to confusion if not explained. All your points hold, to my understanding. (Now I'm concerned, in fact, that perhaps my answer doesn't serve to clarify, as I had hoped, but merely confuses the issue. Do you think there's a suitable modification that would help it? Should I delete it?) $\endgroup$ – Glen_b -Reinstate Monica Feb 21 '13 at 14:05
  • $\begingroup$ I would hate to see it deleted, Glen. If you intend to make changes, consider more explicitly pointing out which aspects of this issue you are writing about (e.g., what precisely you mean by a "given model" and what you have in mind about models with "different characteristics"). This was the spirit (collaborative, not critical) in which I offered my comment. $\endgroup$ – whuber Feb 21 '13 at 15:13
  • $\begingroup$ I didn't feel criticized by you - you seemed to be clarifying and nothing more - but the need for it highlights an inadequacy in the answer I had been concerned about before you commented. The vagueness of 'different characteristics' are because it's a pretty general thing - vary much of anything (I even give examples of something as simple as removing a point or adding a variable to illustrate how little one needs to change) can make that monotonic relationship evaporate. I'll think about what more I might say. $\endgroup$ – Glen_b -Reinstate Monica Feb 21 '13 at 16:15
  • $\begingroup$ +1 for the edit: these are valuable comments and it's especially useful to see the formula for $F$ appear. $\endgroup$ – whuber Feb 21 '13 at 22:24
3
$\begingroup$

"for OLS regression, does a higher R-squared also imply a higher P-value? Specifically for a single explanatory variable (Y = a + bX + e) "

Specifically for a single explanatory variable, given the sample size, the answer is yes. As Glen_b has explained, there is a direct relationship between $R^2$ and the test statistic (be it a $F$ or $t$). For instance, as explained in this other question (High $R^2$ squared and high $p$-value for simple linear regression) for the simple linear regression with one covariate (and a constant), the relationship between $t$ and $R^2$ is:

$|t| = \sqrt{\frac{R^2}{(1- R^2)}(n -2)}$

So in this case, once you fix $n$, the higher the $R^2$ the higher the $t$ statistic and the lower the p-value.

"but would also be interested to know for n multiple explanatory variables (Y = a + b1X + ... bnX + e)."

The answer is the same, but instead of looking at one variable only, we now look at all variables together -- hence the $F$ statistic, as Glen_b has shown. And here you have to fix both $n$ and the number of parameters. Or, to put it better, fix the degrees of freedom.

Context - I'm performing OLS regression on a range of variables and am trying to develop the best explanatory functional form (...)

Ok, so this is actually a different problem. If you are looking at the best explanatory functional form, you should also take a look at cross-validation techniques. Even if $R^2$ is the quantity of interest for your problem (it usually isn't), finding the best fit in-sample can be very misleading -- you usually want your findings to generalize out of sample, and proper cross-validation can help you not overfit your data too much.

And here I'm guessing that you want "predictive" power (since you say you want to find "the best explanatory functional form"). If you want to do causal inference, for instance, then the $R^2$ or other predictive performance metrics are of little help without more structural/substantive knowledge of the problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.