# Should I fit regression line to scatter or to the mean values?

I have two random variables and I am trying to see if and how much they are correlated. One of them (say X) is discrete and the other (say Y) is continuous. I used Seaborn to do the linear regression on the scatter plot for X-Y and I got the following:

In this case, the correlation between the two comes out tp be 0.5. I then took mean of all Y values for a given X and then did the regression again and got this:

Now I get the correlation to be close to 1! My question is, which one of these provides realistic description of the datasets and why?

• Actually the situation may be even more complicated than I assumed in my comments to @AdamO - it's not just that the conditional mean is not linear in dist. Apparently even the distribution "around" the conditional mean seems to be a function of dist. Maybe you need a GLM. I suggest that you ask another question where first of all you describe in much more detail your problem (what are assort and dist? How are they measured? What is your hypothesis on the relationship between them?) and then you ask which model would be the more appropriate for this problem. Jun 16, 2017 at 22:19

## 2 Answers

The averages may be close to linearly related but individual points are not.

There is variation from a linear relationship. A correlation coefficient can show this in two ways. 1) if there is scatter (which is deviation from linear). 2) if the relationship is not linear (which is, clearly, deviation from linear).

What you found out is that the function describing the relationship of the averages is pretty much linear (your second analysis, which shows that the 'line' versus 'averages of multiple points', is nearly the same model), but the linear relation has a lot of scatter on top of it (the first analysis). This means that, the two coordinates for a point are not strongly linearly related.

The correlation tells how much, in the case for 'any point', one variable can be (linearly) related to another one. Your first image clearly shows that this is not the case (or at least not so much, since you end up in the middle between 0 and 1).

The best image to show the relationship would be the combination of both images. In that way another observer could directly see what is going on (which is two things at the same time, there is a linear trend, but still there is also some, non uniform, scatter). I would plot the averages, or maybe boxplots, with estimates and the data points on top of it (some jittering would be nice, it is currently difficult to see how many points you have in your 4 and 5 'dist' levels). Although this can be done in multiple ways. It depends on what your readers would like to see, or can handle.

An interesting point is to see that the lines are not the same in the two different cases. This is because the different weights. In the second image each average counts the same weight. In the first image this is not the case. It seems to me that there are four heavy points, the three at assort=0 and the one at dist = 4 and assort = 0.75, which dominate the line.

It looks like the linear relation ship is mostly in the variation of the scatter among the different 'dist' levels. It might be interesting to see what happens with the median or with other percentile levels (you chose the mean in your second study, but you may wonder whether it shows correctly what is going on). Also showing the variation of the scatter (and not just the means) seems to me a correct way to present the data.

That is simply regression to the mean! The averages will always be less disperse, less heterogeneous than the specific observations.

In a regression model, however, it is good to assess the adequacy of the straight-line approximation obtained in linear regression. Calculating averages over the domain of one or more regressors is a very good exploratory analysis, but I would not compare the sample correlation to the aggregate correlation because there will be regression to the mean.

• Just now somebody asked me to build a Bayesian model because fluctuations are not Gaussian. Do you have any good reference for that? Jun 16, 2017 at 17:14
• @Peaceful that is a separate question, and probably too broad for this site without more specific details. It is a known fact to statisticians that linear regression is perfectly suited to modeling conditional means if "fluctuations" (errors) are not Gaussian. Jun 16, 2017 at 17:46
• @AdamO well, it depends on what method you use to estimate the linear model. If you use OLS and errors are not normal, then inference is not nearly perfect, including confidence intervals and $p$-values, which the OP seems to care about. Now I don't know Seaborn and I can't be quite sure which estimator it implements, but I would be frankly quite surprised if it had anything "fancier" than OLS, such as for example the Huber $M-$estimator or the sandwich estimator. Jun 16, 2017 at 21:43
• @Peaceful the point is not just that the errors are (evidently) non-Gaussian. It's also quite evident that the conditional mean is not a linear function of dist (btw, dist is ordinal, not just categorical, right?), since the response appears to have an upper bound. It's not easy to say since you have much more observations at values of dist < 4, than you have for dist equal to 4 or 5, but I think you have sufficient elements to rule out the possibility that the conditional mean is linear in dist. Jun 16, 2017 at 21:52
• @AdamO: The point is that I have never heard of Building a Gaussian model for regression! Can you suggest something to start with? Jun 17, 2017 at 6:07