Precise meaning of and comparison between influential point, high leverage point, and outlier?

Question

From Wikipedia

Influential observations are those observations that have a relatively large effect on the regression model's predictions.

From Wikipedia

Leverage points are those observations, if any, made at extreme or outlying values of the independent variables such that the lack of neighboring observations means that the fitted regression model will pass close to that particular observation.

Why is the following comparison from Wikipedia

Although an influential point will typically have high leverage, a high leverage point is not necessarily an influential point.

Answer 1

Imagine any regression line fitted to some data.

Now imagine an extra data point, an outlier some distance away from the main body of the data, but one which lies somewhere along that regression line.

If the regression line were to be refitted, the coefficients would not change. Conversely, deleting the extra outlier would have zero influence on the coefficients.

So, an outlier or leverage point would have zero influence if it were perfectly consistent with the rest of the data and the model that rest implies.

For "line" read "plane" or "hyperplane" if desired, but the simplest example of two variables and a scatter plot is enough here.

However, as you are fond of definitions -- often, it seems, tending to read too much into them -- here is my favourite definition of outliers:

"Outliers are sample values that cause surprise in relation to the majority of the sample" (W.N. Venables and B.D. Ripley. 2002. Modern applied statistics with S. New York: Springer, p.119).

Crucially, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say if the data really are lognormal or gamma rather than normal.

P.S. I don't think that leverage points necessarily lack neighbouring observations. For example, they may occur in pairs.

Answer 2

It's easy to illustrate how a high leverage point might not be influential in the case of a simple linear model:

The blue line is a regression line based on all the data, the red line ignores the point at the top right of the plot.

This point fits the definition of a high leverage point you just provided as it is far away from the rest of the data. Because of that, the regression line (the blue one) has to pass close to it. But since its position largely fits the pattern observed in the rest of the data, the other model would predict it very well (i.e. the red line already passes close to it in any case) and it is therefore not particularly influential.

Compare this to the following scatterplot:

Here, the point on the right of the plot is still a high leverage point but this time it does not really fit the pattern observed in the rest of the data. The blue line (the linear fit based on all the data) passes very close but the red line does not. Including or excluding this one point changes the parameter estimates dramatically: It has a lot of influence.

Note that the definitions you cited and the examples I just gave might seem to imply that high leverage/influential points are, in some sense, univariate “outliers” and that the fitted regression line will pass close to points with the highest influence but it need not be the case.

In this last example, the observation on the bottom right has a (relatively) large effect on the fit of the model (visible again through the difference between the red and the blue lines) but it still appears to be far away from the regression line while being undetectable in univariate distributions (represented here by the “rugs” along the axes).

Answer 3

Great intuitive and visual answers above, let me add a little formulas in the case of linear regression in 2 dimensions. The predictor at a given data point $x_i$ is $\hat{y} = \beta_0 + \beta_1 x_i$.

Being high leverage concern only $x_i$, $|x_i- \bar{x}|$ is unusually large compare to average. Meanwhile, being outlier concerns $y_i$, when the residual $|\hat{y}-y_i|$ given $x_i$ is unusually large.

For the same residual $|\hat{y}-y|$, a high leverage point has a stronger influence to what $\beta_1$ best fits the data to minimize $\sum_i^N (\beta_0 + \beta_1 x_i-y)^2$ because the way it forms a product with $\beta_1$ in the predictor. My mental picture is that a point far away from the data mean has a "longer lever" to push the slope than a point near the mean.