414
$\begingroup$

Suppose we have data set $(X_i,Y_i)$ with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(X_i,Y_j)$. Is there any meaningful interpretation of the regression on the new data set? Does this have a name?

I imagine this is a silly question so I apologize, I'm not formally trained in statistics. In my mind this completely destroys our data and the regression is meaningless. But my manager says he gets "better regressions most of the time" when he does this (here "better" means more predictive). I have a feeling he is deceiving himself.

EDIT: Thank you for all of your nice and patient examples. I showed him the examples by @RUser4512 and @gung and he remains staunch. He's becoming irritated and I'm becoming exhausted. I feel crestfallen. I will probably begin looking for other jobs soon.

$\endgroup$
38
  • 161
    $\begingroup$ But my manager says he gets "better regressions most of the time" when he does this. Oh god... $\endgroup$ Commented Dec 7, 2015 at 17:30
  • 71
    $\begingroup$ There's certainly no reason for you to feel embarrassed! $\endgroup$ Commented Dec 7, 2015 at 19:10
  • 48
    $\begingroup$ "Science is whatever we want it to be." - Dr. Leo Spaceman. $\endgroup$
    – Sycorax
    Commented Dec 7, 2015 at 19:31
  • 63
    $\begingroup$ This idea has to compete with another I have encountered: If your sample is small, just bulk it up with several copies of the same data. $\endgroup$
    – Nick Cox
    Commented Dec 7, 2015 at 20:11
  • 55
    $\begingroup$ You should tell your boss you have a better idea. Instead of using the actual data just generate your own because it'll be easier to model. $\endgroup$
    – dsaxton
    Commented Dec 7, 2015 at 20:13

17 Answers 17

186
$\begingroup$

I'm not sure what your boss thinks "more predictive" means. Many people incorrectly believe that lower $p$-values mean a better / more predictive model. That is not necessarily true (this being a case in point). However, independently sorting both variables beforehand will guarantee a lower $p$-value. On the other hand, we can assess the predictive accuracy of a model by comparing its predictions to new data that were generated by the same process. I do that below in a simple example (coded with R).

options(digits=3)                       # for cleaner output
set.seed(9149)                          # this makes the example exactly reproducible

B1 = .3
N  = 50                                 # 50 data
x  = rnorm(N, mean=0, sd=1)             # standard normal X
y  = 0 + B1*x + rnorm(N, mean=0, sd=1)  # cor(x, y) = .31
sx = sort(x)                            # sorted independently
sy = sort(y)
cor(x,y)    # [1] 0.309
cor(sx,sy)  # [1] 0.993

model.u = lm(y~x)
model.s = lm(sy~sx)
summary(model.u)$coefficients
#             Estimate Std. Error t value Pr(>|t|)
# (Intercept)    0.021      0.139   0.151    0.881
# x              0.340      0.151   2.251    0.029  # significant
summary(model.s)$coefficients
#             Estimate Std. Error t value Pr(>|t|)
# (Intercept)    0.162     0.0168    9.68 7.37e-13
# sx             1.094     0.0183   59.86 9.31e-47  # wildly significant

u.error = vector(length=N)              # these will hold the output
s.error = vector(length=N)
for(i in 1:N){
  new.x      = rnorm(1, mean=0, sd=1)   # data generated in exactly the same way
  new.y      = 0 + B1*x + rnorm(N, mean=0, sd=1)
  pred.u     = predict(model.u, newdata=data.frame(x=new.x))
  pred.s     = predict(model.s, newdata=data.frame(x=new.x))
  u.error[i] = abs(pred.u-new.y)        # these are the absolute values of
  s.error[i] = abs(pred.s-new.y)        #  the predictive errors
};  rm(i, new.x, new.y, pred.u, pred.s)
u.s = u.error-s.error                   # negative values means the original
                                        # yielded more accurate predictions
mean(u.error)  # [1] 1.1
mean(s.error)  # [1] 1.98
mean(u.s<0)    # [1] 0.68


windows()
  layout(matrix(1:4, nrow=2, byrow=TRUE))
  plot(x, y,   main="Original data")
  abline(model.u, col="blue")
  plot(sx, sy, main="Sorted data")
  abline(model.s, col="red")
  h.u = hist(u.error, breaks=10, plot=FALSE)
  h.s = hist(s.error, breaks=9,  plot=FALSE)
  plot(h.u, xlim=c(0,5), ylim=c(0,11), main="Histogram of prediction errors",
       xlab="Magnitude of prediction error", col=rgb(0,0,1,1/2))
  plot(h.s, col=rgb(1,0,0,1/4), add=TRUE)
  legend("topright", legend=c("original","sorted"), pch=15, 
         col=c(rgb(0,0,1,1/2),rgb(1,0,0,1/4)))
  dotchart(u.s, color=ifelse(u.s<0, "blue", "red"), lcolor="white",
           main="Difference between predictive errors")
  abline(v=0, col="gray")
  legend("topright", legend=c("u better", "s better"), pch=1, col=c("blue","red"))

enter image description here

The upper left plot shows the original data. There is some relationship between $x$ and $y$ (viz., the correlation is about $.31$.) The upper right plot shows what the data look like after independently sorting both variables. You can easily see that the strength of the correlation has increased substantially (it is now about $.99$). However, in the lower plots, we see that the distribution of predictive errors is much closer to $0$ for the model trained on the original (unsorted) data. The mean absolute predictive error for the model that used the original data is $1.1$, whereas the mean absolute predictive error for the model trained on the sorted data is $1.98$—nearly twice as large. That means the sorted data model's predictions are much further from the correct values. The plot in the lower right quadrant is a dot plot. It displays the differences between the predictive error with the original data and with the sorted data. This lets you compare the two corresponding predictions for each new observation simulated. Blue dots to the left are times when the original data were closer to the new $y$-value, and red dots to the right are times when the sorted data yielded better predictions. There were more accurate predictions from the model trained on the original data $68\%$ of the time.


The degree to which sorting will cause these problems is a function of the linear relationship that exists in your data. If the correlation between $x$ and $y$ were $1.0$ already, sorting would have no effect and thus not be detrimental. On the other hand, if the correlation were $-1.0$, the sorting would completely reverse the relationship, making the model as inaccurate as possible. If the data were completely uncorrelated originally, the sorting would have an intermediate, but still quite large, deleterious effect on the resulting model's predictive accuracy. Since you mention that your data are typically correlated, I suspect that has provided some protection against the harms intrinsic to this procedure. Nonetheless, sorting first is definitely harmful. To explore these possibilities, we can simply re-run the above code with different values for B1 (using the same seed for reproducibility) and examine the output:

  1. B1 = -5:

    cor(x,y)                            # [1] -0.978
    summary(model.u)$coefficients[2,4]  # [1]  1.6e-34  # (i.e., the p-value)
    summary(model.s)$coefficients[2,4]  # [1]  1.82e-42
    mean(u.error)                       # [1]  7.27
    mean(s.error)                       # [1] 15.4
    mean(u.s<0)                         # [1]  0.98
    
  2. B1 = 0:

    cor(x,y)                            # [1] 0.0385
    summary(model.u)$coefficients[2,4]  # [1] 0.791
    summary(model.s)$coefficients[2,4]  # [1] 4.42e-36
    mean(u.error)                       # [1] 0.908
    mean(s.error)                       # [1] 2.12
    mean(u.s<0)                         # [1] 0.82
    
  3. B1 = 5:

    cor(x,y)                            # [1] 0.979
    summary(model.u)$coefficients[2,4]  # [1] 7.62e-35
    summary(model.s)$coefficients[2,4]  # [1] 3e-49
    mean(u.error)                       # [1] 7.55
    mean(s.error)                       # [1] 6.33
    mean(u.s<0)                         # [1] 0.44
    
$\endgroup$
4
  • 18
    $\begingroup$ Your answer makes a very good point, but perhaps not as clearly as it could and should. It's not necessarily obvious to a layperson (like, say, the OP's manager) what all those plots at the end (never mind the R code) actually show and imply. IMO, your answer could really use an explanatory paragraph or two. $\endgroup$ Commented Dec 7, 2015 at 19:57
  • 4
    $\begingroup$ Thanks for your comment, @IlmariKaronen. Can you suggest things to add? I tried to make the code as self-explanatory as possible, & commented it extensively. But I may no longer be able to see these things with the eyes of someone who isn't familiar w/ these topics. I will add some text to describe the plots at the bottom. If you can think of anything else, please let me know. $\endgroup$ Commented Dec 7, 2015 at 20:02
  • 18
    $\begingroup$ +1 This still is the sole answer that addresses the situation proposed: when two variables already exhibit some positive association, it nevertheless is an error to regress the independently sorted values. All the other answers assume there is no association or that it is actually negative. Although they are good examples, since they don't apply they won't be convincing. What we still lack, though, is a gut-level intuitive real-world example of data like those simulated here where the nature of the mistake is embarrassingly obvious. $\endgroup$
    – whuber
    Commented Dec 7, 2015 at 23:28
  • $\begingroup$ @dsaxton, I use <- sometimes, but my goal on CV is to write R code as close to pseudocode as possible so that it is more readable for people who aren't familiar w/ R. = is pretty universal among programming languages as an assignment operator. $\endgroup$ Commented Dec 8, 2015 at 18:13
139
$\begingroup$

If you want to convince your boss, you can show what is happening with simulated, random, independent $x,y$ data. With R:enter image description here

n <- 1000

y<- runif(n)
x <- runif(n)

linearModel <- lm(y ~ x)


x_sorted <- sort(x)
y_sorted <- sort(y)

linearModel_sorted <- lm(y_sorted ~ x_sorted)

par(mfrow = c(2,1))
plot(x,y, main = "Random data")
abline(linearModel,col = "red")


plot(x_sorted,y_sorted, main = "Random, sorted data")
abline(linearModel_sorted,col = "red")

Obviously, the sorted results offer a much nicer regression. However, given the process used to generate the data (two independent samples) there is absolutely no chance that one can be used to predict the other.

$\endgroup$
4
  • 17
    $\begingroup$ It is almost like all the Internet "before vs after" advertisements :) $\endgroup$
    – Tim
    Commented Dec 7, 2015 at 19:19
  • $\begingroup$ This is a good example, but it don't think it will convince him because our data does have positive correlation before sorting. Sorting just "reinforces" the relationship (albeit an incorrect one). $\endgroup$ Commented Dec 7, 2015 at 19:22
  • 23
    $\begingroup$ @arbitraryuser: Well, sorted data will always show a positive (well, non-negative) correlation, no matter what, if any, correlation the original data had. If you know that the original data always has a positive correlation anyway, then it's "correct by accident" -- but then, why even bother checking for correlation, if you already know it's present and positive anyway? The test your manager is running is a bit like an "air quality detector" that always says "breathable air detected" -- it works perfectly, as long as you never take it anyplace where there isn't breathable air. $\endgroup$ Commented Dec 7, 2015 at 19:53
  • 3
    $\begingroup$ @arbitraryuser Another example you might find more persuasuve is to take x=0:50, and y=0:-50, a perfect line with slope -1. If you sort them, the relationship turns into a perfect line with slope 1. If the truth is that your variables vary in perfect opposition, and you make a policy prescription based on your mistaken perception that they vary in perfect agreement, you'll be doing exactly the wrong thing. $\endgroup$ Commented Jan 23, 2019 at 22:33
110
$\begingroup$

Your intuition is correct: the independently sorted data have no reliable meaning because the inputs and outputs are being randomly mapped to one another rather than what the observed relationship was.

There is a (good) chance that the regression on the sorted data will look nice, but it is meaningless in context.

Intuitive example: Suppose a data set $(X = age, Y = height)$ for some population. The graph of the unadulterated data would probably look rather like a logarithmic or power function: faster growth rates for children that slow for later adolescents and "asymptotically" approach one's maximum height for young adults and older.

If we sort $x, y$ in ascending order, the graph will probably be nearly linear. Thus, the prediction function is that people grow taller for their entire lives. I wouldn't bet money on that prediction algorithm.

$\endgroup$
6
  • 26
    $\begingroup$ +1--but I would drop the "essentially" and re-emphasize the "meaningless." $\endgroup$
    – whuber
    Commented Dec 7, 2015 at 17:39
  • 13
    $\begingroup$ Note that the OP refers to independently sorting the data as opposed to shuffling it. This is a subtle but important difference as it pertains to what the observed "relationship" one would see after applying the given operation. $\endgroup$
    – cardinal
    Commented Dec 7, 2015 at 22:06
  • 3
    $\begingroup$ I am confused by the example you added. If $x$ is age and $y$ is height, then both variables are ordered already: nobody's age or height ever decreases. So sorting would not have any effect at all. Cc to @JakeWestfall, who commented that he liked this example. Can you explain? $\endgroup$
    – amoeba
    Commented Dec 8, 2015 at 20:33
  • 14
    $\begingroup$ @amoeba Trivial data set: average teenager, mid-30s NBA center, elderly average woman. After sorting the prediction algorithm is that the oldest is the tallest. $\endgroup$
    – d0rmLife
    Commented Dec 8, 2015 at 21:12
  • 1
    $\begingroup$ @amoeba I see how it could be interpreted like that, I will clarify. $\endgroup$
    – d0rmLife
    Commented Dec 8, 2015 at 22:12
55
$\begingroup$

Actually, let's make this really obvious and simple. Suppose I conduct an experiment in which I measure out 1 liter of water in a standardized container, and I look at the amount of water remaining in the container $V_i$ as a function of time $t_i$, the loss of water due to evaporation:

Now suppose I obtain the following measurements $(t_i, V_i)$ in hours and liters, respectively: $$(0,1.0), (1,0.9), (2,0.8), (3,0.7), (4,0.6), (5,0.5).$$ This is quite obviously perfectly correlated (and hypothetical) data. But if I were to sort the time and the volume measurements, I would get $$(0,0.5), (1,0.6), (2,0.7), (3,0.8), (4,0.9), (5,1.0).$$ And the conclusion from this sorted data set is that as time increases, the volume of water increases, and moreover, that starting from 1 liter of water, you would get after 5 hours of waiting, more than 1 liter of water. Isn't that remarkable? Not only is the conclusion opposite of what the original data said, it also suggests we have discovered new physics!

$\endgroup$
1
  • 8
    $\begingroup$ Nice intuitive example! Except for the last line. With the original data we would get a negative volume after time, which is just as well new physics. You can't ever really extrapolate a regression. $\endgroup$
    – Jongsma
    Commented Dec 8, 2015 at 15:06
29
$\begingroup$

It is a real art and takes a real understanding of psychology to be able to convince some people of the error of their ways. Besides all the excellent examples above, a useful strategy is sometimes to show that a person's belief leads to an inconsistency with herself. Or try this approach. Find out something your boss believes strongly about such as how persons perform on task Y has no relation with how much of an attribute X they possess. Show how your boss's own approach would result in a conclusion of a strong association between X and Y. Capitalize on political/racial/religious beliefs.

Face invalidity should have been enough. What a stubborn boss. Be searching for a better job in the meantime. Good luck.

$\endgroup$
17
$\begingroup$

This technique is actually amazing. I'm finding all sorts of relationships that I never suspected. For instance, I would have not have suspected that the numbers that show up in Powerball lottery, which it is CLAIMED are random, actually are highly correlated with the opening price of Apple stock on the same day! Folks, I think we're about to cash in big time. :)

> powerball_last_number = scan()
1: 69 66 64 53 65 68 63 64 57 69 40 68
13: 
Read 12 items
> #Nov. 18, 14, 11, 7, 4
> #Oct. 31, 28, 24, 21, 17, 14, 10
> #These are powerball dates.  Stock opening prices 
> #are on same or preceding day.
> 
> appl_stock_open = scan()
1: 115.76  115.20 116.26  121.11  123.13 
6: 120.99  116.93  116.70  114.00  111.78
11: 111.29  110.00
13: 
Read 12 items
> hold = lm(appl_stock_open ~ powerball_last_number)
> summary(hold)


Coefficients:
                       Estimate Std. Error t value Pr(>|t|)    
(Intercept)           112.08555    9.45628  11.853 3.28e-07 ***
powerball_last_number   0.06451    0.15083   0.428    0.678    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.249 on 10 degrees of freedom
Multiple R-squared:  0.01796,   Adjusted R-squared:  -0.08024 
F-statistic: 0.1829 on 1 and 10 DF,  p-value: 0.6779

Hmm, doesn't seem to have a significant relationship. BUT using the new, improved technique:

> 
> vastly_improved_regression = lm(sort(appl_stock_open)~sort(powerball_last_number))
> summary(vastly_improved_regression)

Coefficients:
                            Estimate Std. Error t value Pr(>|t|)    
(Intercept)                 91.34418    5.36136  17.038 1.02e-08 ***
sort(powerball_last_number)  0.39815    0.08551   4.656    9e-04 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.409 on 10 degrees of freedom
Multiple R-squared:  0.6843,    Adjusted R-squared:  0.6528 
F-statistic: 21.68 on 1 and 10 DF,  p-value: 0.0008998

NOTE: This is not meant to be a serious analysis. Just show your manager that they can make ANY two variables significantly related if you sort them both.

$\endgroup$
16
$\begingroup$

One more example. Imagine that you have two variables, one connected with eating chocolate and second one connected to overall well-being. You have a sample of two and your data looks like below:

$$ \begin{array}{cc} \text{chocolate} & \text{no happiness} \\ \text{no chocolate} & \text{happiness} \\ \end{array} $$

What is the relation of chocolate and happiness based on your sample? And now, change order of one of the columns - what is the relation after this operation?

The same problem can be approached differently. Say, that you have a bigger sample, with some number of cases and you measure two continuous variables: chocolate consumption per day (in grams) and happiness (imagine that you have some way to measure it). If you are interested if they are related you can measure correlation or use linear regression model, but sometimes in such cases people simply dichotomize one variable and use it as a grouping factor with $t$-test (this is not the best and not recommended approach, but let me use it as an example). So you divide your sample into two groups: with high chocolate consumption and with low chocolate consumption. Next, you compare average happiness in both groups. Now imagine what would happen if you sorted happiness variable independently of grouping variable: all the cases with high happiness would go go high chocolate consumption group, and all the low happiness cases would end up in low chocolate consumption group -- would such hypothesis test have any sens? This can be easily extrapolated into regression if you imagine that instead of two groups for chocolate consumption you have $N$ such groups, one for each participant (notice that $t$-test is related to regression).

In bivariate regression or correlation we are interested in pairwise relations between each $i$-th value of $X$ and $i$-th value of $Y$, changing order of the observations destroys this relation. If you sort both variables that this always leads them to be more positively correlated with each other since it will always be the case that if one of the variables increases, the other one also increases (because they are sorted!).

Notice that sometimes we actually are interested in changing order of cases, we do so in resampling methods. For example, we can intentionally shuffle observations multiple times so to learn something about null distribution of our data (how would our data look like if there was no pairwise relations), and next we can compare if our real data is anyhow better than the randomly shuffled. What your manager does is exactly the opposite -- he intentionally forces the observations to have artificial structure where there was no structure, what leads to bogus correlations.

$\endgroup$
0
9
$\begingroup$

A simple example that maybe your manager could understand:

Let's say you have Coin Y and Coin X, and you flip each of them 100 times. Then you want to predict whether getting a heads with Coin X (IV) can increase the chance of getting a heads with Coin Y (DV).

Without sorting, the relationship will be none, because Coin X's outcome shouldn't affect the Coin Y's outcome. With sorting, relationship will be nearly perfect.

How does it make sense to conclude that you have a good chance of getting a heads on a coin flip if you have just flipped a heads with a different coin?

$\endgroup$
0
8
$\begingroup$

Plenty of good counter examples in here. Let me just add a paragraph about the heart of the problem.

You are looking for a correlation between $X_i$ and $Y_i$. That means that $X$ and $Y$ both tend to be large for the same $i$ and small for the same $i$. So a correlation is a property of $X_1$ linked with $Y_1$, $X_2$ linked with $Y_2$, and so on. By sorting $X$ and $Y$ independently you (in most cases) lose the pairing. $X_1$ will no longer be paired up with $Y_1$. So the correlation of the sorted values will not measure the connection between $X_1$ and $Y_1$ that you are after.

Actually, let me add a paragraph about why it "works" as well.

When you sort both lists, let's call the new sorted list $X_a$, $X_b$, and so on, $X_a$ will be smallest $X$ value, and $Y_a$ will be the smallest Y value. $X_z$ will be the largest $X$ and $Y_z$ will be the largest $Y$. Then you query the new lists if small and large value co occur. That is, you ask if $X_a$ is small when $Y_a$ is small. Is $X_z$ large when $Y_z$ is large? Of course the answer is yes, and of course we will get almost perfect correlation. Does that tell you anything about $X_1$'s relationship with $Y_1$? No.

$\endgroup$
8
$\begingroup$

Actually, the test that is described (i.e. sort the X values and the Y values independently and regress one against the other) DOES test something, assuming that the (X,Y) are sampled as independent pairs from a bivariate distribution. It just isn't a test of what your manager wants to test. It is essentially checking the linearity of a QQ-plot, comparing the marginal distribution of the Xs with the marginal distribution of the Ys. In particular, the 'data' will fall close to a straight line if the density of the Xs (f(x)) is related to the density of the Ys (g(y)) this way:

$f(x) = g((y-a)/b)$ for some constants $a$ and $b>0$. This puts them in a location-scale family. Unfortunately this is not a method to get predictions...

$\endgroup$
8
$\begingroup$

Strange that the most obvious counterexample is still not present among the answers in its simplest form.

Let $Y = -X$.

If you sort the variables separately and fit a regression model on such data, you should obtain something like $\hat Y \approx X$ (because when the variables are sorted, larger values of one must correspond to larger values of the other).

This is a kind-of a "direct inverse" of the pattern you might be willing to find here.

$\endgroup$
8
  • $\begingroup$ Could you explain what assertion this is a counterexample to? $\endgroup$
    – whuber
    Commented May 17, 2017 at 18:31
  • $\begingroup$ The assertion of the manager that you can "get better regressions all the time" by sorting inputs and outputs independently. $\endgroup$
    – KT.
    Commented May 18, 2017 at 13:18
  • $\begingroup$ Thank you. I don't see why your example disproves that, though: in both cases $R^2=1$, so the regressions are equally "good". $\endgroup$
    – whuber
    Commented May 18, 2017 at 13:34
  • $\begingroup$ Try measuring this $R^2$ on a hold-out set. $\endgroup$
    – KT.
    Commented May 19, 2017 at 9:29
  • 1
    $\begingroup$ Also note that I find it strange that you seem to misunderstand my example while ignoring all the other answers here. All of them are showing examples of models which would be fit incorrectly using the "sorting" approach, despite the fact of probably having a better $R^2$ on the training set if sorted. I just thought that considering the $Y = -X$ may be more intuitive than most other examples here for its simplicity and obvious mismatch of the results you obtain. $\endgroup$
    – KT.
    Commented May 19, 2017 at 9:33
6
$\begingroup$

Let me play Devil's Advocate here. I think many answers have made convincing cases that the boss' procedure is fundamentally mistaken. At the same time, I offer a counter-example that illustrates that the boss may have actually seen results improve with this mistaken transformation.

I think that acknowledging that this procedure might've "worked" for the boss could begin a more-persuasive argument: Sure, it did work, but only under these lucky circumstances that usually won't hold. Then we can show -- as in the excellent accepted answer -- how bad it can be when we're not lucky. Which is most of the time. In isolation, showing the boss how bad it can be might not persuade him because he might have seen a case where it does improve things, and figure that our fancy argument must have a flaw somewhere.

I found this data online, and sure enough, it appears that the regression is improved by the independent sorting of X and Y because: a) the data is highly positively correlated, and b) OLS really doesn't do well with extreme (high-leverage) outliers. The height and weight have a correlation of 0.19 with the outlier included, 0.77 with the outlier excluded, and 0.78 with X and Y independently sorted.

x <- read.csv ("https://vincentarelbundock.github.io/Rdatasets/csv/car/Davis.csv", header=TRUE)

plot (weight ~ height, data=x)

lm1 <- lm (weight ~ height, data=x)

xx <- x
xx$weight <- sort (xx$weight)
xx$height <- sort (xx$height)

plot (weight ~ height, data=xx)

lm2 <- lm (weight ~ height, data=xx)

plot (weight ~ height, data=x)
abline (lm1)
abline (lm2, col="red")

enter image description here

plot (x$height, x$weight)
points (xx$height, xx$weight, col="red")

enter image description here

So it appears to me that the regression model on this dataset is improved by the independent sorting (black versus red line in first graph), and there is a visible relationship (black versus red in the second graph), due to the particular dataset being highly (positively) correlated and having the right kind of outliers that harm the regression more than the shuffling that occurs when you independently sort x and y.

Again, not saying independently sorting does anything sensible in general, nor that it's the correct answer here. Just that the boss might have seen something like this that happened to work under just the right circumstances.

$\endgroup$
4
  • 1
    $\begingroup$ It looks like a pure coincidence that you arrived at similar correlation coefficients. This example does not appear to demonstrate anything about a relationship between the original and independently-sorted data. $\endgroup$
    – whuber
    Commented May 17, 2017 at 18:30
  • 2
    $\begingroup$ @whuber: How about the second graph? It feels to me that if the original data is highly correlated, sorting them may only shuffle values a bit, basically preserving the original relationship +/-. With a couple of outliers, things get rearranged more, but... Sorry I don't have the math chops to go farther than that. $\endgroup$
    – Wayne
    Commented May 17, 2017 at 18:39
  • 1
    $\begingroup$ I think the intuition you express is correct, Wayne. The logic of the question--as I interpret it--concerns what you can say about the original data based on the scatterplot of the sorted variables alone. The answer is, absolutely nothing beyond what you can infer from their separate (univariate) distributions. The point is that the red dots in your second graph are consistent not only with the data you show, but also with all the astronomically huge number of other permutations of those data--and you have no way of knowing which of those permutations is the right one. $\endgroup$
    – whuber
    Commented May 17, 2017 at 20:35
  • $\begingroup$ @whuber: I've edited the introduction to my answer in a way that I think makes it relevant to the discussion. I think that acknowledging how the boss' procedure might've worked for him could be a first step in a more persuasive argument that jibes with the boss' experience. For your consideration. $\endgroup$
    – Wayne
    Commented May 17, 2017 at 21:13
4
$\begingroup$

It's a QQ-plot, isn't it? You'd use it to compare the distribution of x vs. y. If you'd plot sorted outcomes of a relationship like $x \sim x^2$, the plot would be crooked, which indicates that $x$ and $x^2$ for some sampling of $x$s have different distributions.

The linear regression is usually less reasonable (exceptions exist, see other answers); but the geometry of tails and of distribution of errors tells you how far from similar the distributions are.

$\endgroup$
4
$\begingroup$

Sorting the columns of the following table independently also makes it look "better":

name,country
Alice,DE
Daniel,US
Christian,DE
Bernadette,US

->

name,country
Alice,DE
Bernadette,DE
Christian,US
Daniel,US

Now, all females are from Germany, and all males are from the US.

We can now much more nicely predict the gender by just knowing the country. Isn't that great? /s

$\endgroup$
3
$\begingroup$

You are right. Your manager would find "good" results! But they are meaningless. What you get when you sort them independently is that the two either increase or decrease similarly and this gives a semblance of a good model. But the two variables have been stripped of their actual relationship and the model is incorrect.

$\endgroup$
2
$\begingroup$

I have a simple intuition why this is actually a good idea if the function is monotone:

Imagine you know the inputs $x_1, x_2,\cdots, x_n$ and they are ranked, i.e. $x_i<x_{i+1}$ and assume $f:\Re\mapsto\Re$ is the unknown function we want to estimate. You can define a random model $y_i = f(x_i) + \varepsilon_i$ where $\varepsilon_i$ are independently sampled as follows: $$ \varepsilon_i = f(x_{i+\delta}) - f(x_i) $$ where $\delta$ is uniformly sampled from the discrete set $\{-\Delta,-\Delta+1, \cdots \Delta-1, \Delta\}$. Here, $\Delta\in\mathbb{N}$ controls the variance. For example, $\Delta=0$ gives no noise, and $\Delta=n$ give independent input and outputs.

With this model in mind, the proposed "sorting" method of you boss makes perfect sense: If you rank the data, you somehow reduce this type of noise and the estimation of $f$ should becomes better under mild assumptions.

In fact, a more advanced model would assume that $\varepsilon_i$ are dependent, so that we cannot observe 2 times the same output. In such a case, the sorting method could even be optimal. This might have strong connection with random ranking models, such as Mallow's random permutations.

PS: I find it amazing how an apparently simple question can lead to interesting new ways of re-thinking standards model. Please thank you boss!

$\endgroup$
1
  • 1
    $\begingroup$ How is $x_{i+\delta}$ defined when $i+\delta<1$ or $i+\delta>n$? $\endgroup$ Commented Dec 9, 2015 at 18:45
2
$\begingroup$

Say you have these points on a circle of radius 5. You calculate the correlation:

import pandas as pd
s1 = [(-5, 0), (-4, -3), (-4, 3), (-3, -4), (-3, 4), (0, 5), (0, -5), (3, -4), (3, 4), (4, -3), (4, 3), (5, 0)]
df1 = pd.DataFrame(s1, columns=["x", "y"])
print(df1.corr())

   x  y
x  1  0
y  0  1

Then you sort your x- and y-values and do the correlation again:

s2 = [(-5, -5), (-4, -4), (-4, -4), (-3, -3), (-3, -3), (0, 0), (0, 0), (3, 3), (3, 3), (4, 4), (4, 4), (5, 5)]
df2 = pd.DataFrame(s2, columns=["x", "y"])
print(df2.corr())

   x  y
x  1  1
y  1  1

By this manipulation, you change a data set with 0.0 correlation to one with 1.0 correlation. That's a problem.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.