Outliers change the sign of linear relation

Question

I have a scatter plot with Hgb A1C as the explanatory variable ('x' axis) and FPG ('y' axis)as the result variable. There is a positive linear relationship but it is weak. There is 1 outlier in the y direction. There is 1 outlier in the x direction. When each of these points is removed, r changes in opposite directions. I am trying to explain why this is happening, in simple language.

Would it make sense to say: When the outlier in the y direction is removed, r increases because an outlier that normally falls a distance away from the regression line would decrease the size of the correlation coefficient.

When the outlier in the x direction is removed, r decreases because an outlier that normally falls near the regression line would increase the size of the correlation coefficient.

Does this make sense?

Answer 1

This is usually called the leverage problem; it is easiest to understand in the language of optimization in which regression is done.

Basically, we want to have such a line that sum of squared residuals (distances on y between points and regression line) is minimal; because of that a single point with extreme coordinates may have a dominant contribution to this sum.
Here is an example of such a situation. Green line seems more correct, but its cost is huge because of the outlier's residual (black) -- because of that the red line is generated.

In your case, the line is simply made between those two outliers effectively ignoring or treating the rest of points as one.

Answer 2

What removal of outliers will do to a regression line depends, of course, on where the outliers are. So, rather than use "normally falls" I would be more specific. And it might be clearer if you talk about addition of outliers than removal of them. But I think you are basically correct.