I have a scatter plot with Hgb A1C as the explanatory variable ('x' axis) and FPG ('y' axis)as the result variable. There is a positive linear relationship but it is weak. There is 1 outlier in the y direction. There is 1 outlier in the x direction. When each of these points is removed, r changes in opposite directions. I am trying to explain why this is happening, in simple language.

Would it make sense to say: When the outlier in the y direction is removed, r increases because an outlier that normally falls a distance away from the regression line would decrease the size of the correlation coefficient.

When the outlier in the x direction is removed, r decreases because an outlier that normally falls near the regression line would increase the size of the correlation coefficient.

Does this make sense?

  • $\begingroup$ Yes, I think you're quite right. And overplotting regression lines on four scatterplots (all points, only Y outlier removed, only X outlier removed, and both outliers removed) should illustrate. $\endgroup$ – Assad Ebrahim Oct 6 '12 at 4:16

This is usually called the leverage problem; it is easiest to understand in the language of optimization in which regression is done.

Basically, we want to have such a line that sum of squared residuals (distances on y between points and regression line) is minimal; because of that a single point with extreme coordinates may have a dominant contribution to this sum.
Here is an example of such a situation. Green line seems more correct, but its cost is huge because of the outlier's residual (black) -- because of that the red line is generated.

enter image description here

In your case, the line is simply made between those two outliers effectively ignoring or treating the rest of points as one.


What removal of outliers will do to a regression line depends, of course, on where the outliers are. So, rather than use "normally falls" I would be more specific. And it might be clearer if you talk about addition of outliers than removal of them. But I think you are basically correct.

  • $\begingroup$ Can you please provide an authentic reference for this? $\endgroup$ – Angel Dec 25 '20 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.