4
$\begingroup$

I think my question is somewhat related to this one: What happens if the explanatory and response variables are sorted independently before regression?

But my interest is slightly different. Suppose that I have an original dataset $\{(X_i,Y_i)\}_{i=1}^n$, run OLS of $Y$ on $X$, and calculate MSE and $R^2$.

Now I permute (randomly) the observations of the dependent variable $Y$. Denote the new dataset as $\{(X_i,\tilde{Y}_i)\}_{i=1}^n$.

What is the probability that I will end up with better measures of MSE and $R^2$ when I run OLS of $\tilde{Y}$ on $X$?

Is there a way to bound this probability?

| cite | improve this question | |
$\endgroup$
  • 3
    $\begingroup$ Note that this is the type II error rate (1-power) of a permutation test of the slope; it depends on the underlying slope, the size of the error term (i.e. on $\sigma$) and on the sample size and is different for different distributions (nonparametric tests are distribution free under the null, but generally not under the alternative). If one conditions on the data it might be possible to make some progress with the conditional probability. $\endgroup$ – Glen_b Feb 2 '17 at 2:56
1
$\begingroup$

This is the type II error rate (1-power) of a permutation test of the slope; it depends on the underlying slope, the size of the error term (i.e. on σ) and on the sample size and is different for different distributions (nonparametric tests are distribution free under the null, but generally not under the alternative). If one conditions on the data it might be possible to make some progress with the conditional probability.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I've copied this comment by @Glen_b as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions. $\endgroup$ – mkt - Reinstate Monica Aug 14 '18 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.