Timeline for What is the distribution of $R^2$ in linear regression under the null hypothesis? Why is its mode not at zero when $k>3$?
Current License: CC BY-SA 3.0
38 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 5, 2022 at 21:22 | comment | added | Alecos Papadopoulos | @user1234383 The $n-2$ factor looks to me as a finite-sample or loss-of-degrees-of-freedom correction. | |
| Jul 5, 2022 at 15:10 | comment | added | user1234383 | @AlecosPapadopoulos in the case of a simple regression with one regressor and a constant term (then $k=2$ with your convention) and being the mean of beta function $Beta(a,b)$ equal to $a/(a+b)$ we find that $E(R^2)= 1/(n-1)$. If I am not mistaken, in this specific setup of one regressor + constant, $E(R^2)$ happens to also represent the square of the standard error of the Pearson correlation coefficient which is - under the hypothesis of 0-correlation - 1/(n-2). I was wondering how the two can be reconciled? | |
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Jan 6, 2016 at 14:19 | comment | added | Alecos Papadopoulos | @ChristophHanck See also davegiles.blogspot.jp/2013/05/good-old-r-squared.html | |
| Jan 6, 2016 at 14:18 | comment | added | Alecos Papadopoulos | @ChristophHanck I am not aware of any. This thread, stats.stackexchange.com/q/11553/28746, is tangential to your question. | |
| Jan 6, 2016 at 11:44 | comment | added | Christoph Hanck | +1! Are there results for the distribution of $R^2$ for nonzero $\beta_j$? | |
| Dec 25, 2014 at 15:21 | vote | accept | amoeba | ||
| Dec 24, 2014 at 4:26 | comment | added | Khashaa | @Alecos I beg to differ. In that case "equations" means population moment restrictions, not the effective sample size. So, for classical regression model, the equations are $\mathrm{E}(x_i(y_i-x_i'\beta))=0$. I don't see more equations than unknowns. Anyway, thank you for sharing your wisdom. I like your answer. Especially, associations with spurious regressions. | |
| Dec 24, 2014 at 4:20 | comment | added | Alecos Papadopoulos | @whuber ...which effectively stops it, since I don't use chat. | |
| S Dec 24, 2014 at 4:13 | history | mod moved comments to chat | |||
| S Dec 24, 2014 at 4:13 | comment | added | whuber♦ | Comments are not for extended discussion; this conversation has been moved to chat. | |
| Dec 24, 2014 at 3:59 | comment | added | Alecos Papadopoulos | @Khashaa Except if theory demands it, I never exclude the intercept from the regression -it is the average level of the dependent variable, regressors or no regressors (and this level is usually positive, so it would be a foolishly self-created misspecification to omit it). But I always exclude it from the F-test of the regression, since what I care about is not whether the dependent variable has a non-zero unconditional mean, but whether the regressors have any explanatory power as regards deviations from this mean. | |
| Dec 24, 2014 at 3:41 | comment | added | Khashaa | Of course, most textbooks take the route dispensing with exclusion of intercept with the cautionary tale of dangers using $R^2$ in the absence of intercept. But, be reminded, as Hayashi(2000) put, there are "mixed blessings": it saves you when you mistakenly include all the dummies. | |
| Dec 24, 2014 at 3:20 | comment | added | Alecos Papadopoulos | @Khashaa Testing for a zero intercept amounts to the dependent variable having zero mean. Since in most circumstances regression analysis is done with positive dependent variable, it is not considered in the standard F-test, since it would complicate matters, as you write, without any real gain, or worse, it would tend to reject the null of insignificant regression on account of the non-zero intercept alone. So I chose to treat the standard case producing an answer useful to a wider audience. | |
| Dec 24, 2014 at 3:04 | comment | added | Khashaa | Good analysis, Alecos. Overall, I largely agree with you. I thought the OP asked its distribution under $H_0:\beta_1=\ldots=\beta_k=0$, which certainly involves noncentral $F$ distribution and $R^2$ has a mixture of Beta distributions with Poisson weights. | |
| Dec 23, 2014 at 15:44 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
deleted 2 characters in body
|
| Dec 23, 2014 at 15:21 | comment | added | Alecos Papadopoulos | @amoeba Yes, that's understood, but I updated nevertheless. | |
| Dec 23, 2014 at 15:21 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
added 116 characters in body
|
| Dec 23, 2014 at 15:01 | comment | added | amoeba | Alecos, is @Khashaa correct in saying (in the comments above) that for your answer to be true the null hypothesis should be $\beta_2=...=\beta_k=0$ instead of $\boldsymbol \beta=0$, i.e. intercept can be allowed to be nonzero? | |
| Dec 23, 2014 at 14:58 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
added 15 characters in body
|
| Dec 23, 2014 at 14:03 | comment | added | amoeba | Okay. I edited my question and replaced "has a mode at zero when $k=2$ and $k=3$" with "peaks at zero when $k=2$ and $k=3$". Hope that "peaking at zero" is an informal enough expression to include both cases. | |
| Dec 23, 2014 at 13:58 | comment | added | Alecos Papadopoulos | @amoeba We can define anything we want. But if it is not standard, then it won't be helpful in summarizing information, because it won't be understood. | |
| Dec 23, 2014 at 13:50 | comment | added | amoeba | @Silverfish and Alecos: can't we understand "mode" a bit more broadly and allow singular points to be called a mode, if the distribution goes to $+\infty$ there and nowhere else? Then it would make sense to say that this $R^2$ distribution has a mode ("generalized mode?") at zero for $k=2$. | |
| Dec 23, 2014 at 13:47 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
added 561 characters in body
|
| Dec 23, 2014 at 13:38 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
added 561 characters in body
|
| Dec 23, 2014 at 13:25 | comment | added | Silverfish | @Alecos Excellent answer! (+1) Can I strongly suggest that you add to your answer the requirement for the mode to exist? This is usually stated as $\alpha>1$ and $\beta>1$ but more subtly, it's ok if equality holds in one of the two ... I think for our purposes this becomes $k \geq 3$ and $n \geq k + 2$ and at least one of these inequalities is strict. | |
| Dec 23, 2014 at 13:13 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
deleted 10 characters in body
|
| Dec 23, 2014 at 12:31 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
Added about the mode
|
| Dec 23, 2014 at 12:09 | comment | added | Alecos Papadopoulos | @amoeba This aspect is linked to the phenomenon of spurious fit. | |
| Dec 23, 2014 at 10:29 | comment | added | amoeba | I mean, consider geometrical view of OLS: $\mathbf y$ is a vector in $\mathbb R^n$, $\mathbf X$ defines a $k$-dimensional subspace there. OLS amounts to projecting $\mathbf y$ onto this subspace, and $R^2$ is squared cosine of the angle between $\mathbf y$ and its projection $\hat{\mathbf y}$. Now, from the discussion above it follows that if all vectors are random, then the distribution of this angle will have mode at $90^\circ$ for $k=2$ and $k=3$, but at some other value $< 90^\circ$ for $k>3$. Why?! There should be some geometrical reason here. | |
| Dec 23, 2014 at 10:15 | comment | added | amoeba | Yes, but $R^2$ is obviously not symmetric... Anyway, I think I am going to accept your answer (after some time) because it covers the math, but I continue to be really puzzled about the phase transition between $k=3$ and $k=4$, and would be very interested in understanding this phenomenon. I might offer a bounty for anybody who can provide some intuitive way for looking at it. Perhaps there is a connection to some other phase transitions happening when dimensionality increases. I am quite certain that there should be a geometrical reason here. | |
| Dec 23, 2014 at 10:09 | comment | added | Alecos Papadopoulos | For all cases where the distribution of estimator is symmetric and unimodal, the mode of an ubiased estimator is the true value (and normal and t- distributions fall to this category). In general look up MAPE perhaps, in a Bayesian context, en.wikipedia.org/wiki/Maximum_a_posteriori_estimation | |
| Dec 23, 2014 at 10:03 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
added 264 characters in body
|
| Dec 23, 2014 at 9:53 | comment | added | Alecos Papadopoulos | Its mathematical. For $k=2$ the first parameter of the beta distribution (the "$\alpha$" in standard notation) becomes smaller than unity. In that case the Beta distribution has no finite mode, play around with keisan.casio.com/exec/system/1180573226 to see how the shapes change. | |
| Dec 23, 2014 at 9:45 | comment | added | amoeba | Thank you, @Alecos, very nice. I have a technical and a conceptual question. Technical: formula $\frac{k-3}{n-5}$ for the mode seems to break down when $k=2$. I think it means that $\lim_{x\to 0} = \infty$, so in some sense zero is still the mode. Is that correct? (Not sure what happens when $k=1$ though). Conceptual: so in reasonable cases of $n \gg 1$ we have that the mode is at zero for $k=2$ and $k=3$, but not at zero for $k>3$. Do you have any intuition for this "phase transition"? | |
| Dec 23, 2014 at 3:22 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
deleted 207 characters in body
|
| Dec 23, 2014 at 3:14 | history | edited | Alecos Papadopoulos | CC BY-SA 3.0 |
added 553 characters in body
|
| Dec 23, 2014 at 2:59 | history | answered | Alecos Papadopoulos | CC BY-SA 3.0 |