# Precise meaning of and comparison between influential point, high leverage point, and outlier?

From Wikipedia

Influential observations are those observations that have a relatively large effect on the regression model's predictions.

From Wikipedia

Leverage points are those observations, if any, made at extreme or outlying values of the independent variables such that the lack of neighboring observations means that the fitted regression model will pass close to that particular observation.

Why is the following comparison from Wikipedia

Although an influential point will typically have high leverage, a high leverage point is not necessarily an influential point.

• The answers below are good. It may also help to read my answer here: Interpreting plot.lm(). Nov 13, 2014 at 18:35

Imagine any regression line fitted to some data.

Now imagine an extra data point, an outlier some distance away from the main body of the data, but one which lies somewhere along that regression line.

If the regression line were to be refitted, the coefficients would not change. Conversely, deleting the extra outlier would have zero influence on the coefficients.

So, an outlier or leverage point would have zero influence if it were perfectly consistent with the rest of the data and the model that rest implies.

For "line" read "plane" or "hyperplane" if desired, but the simplest example of two variables and a scatter plot is enough here.

However, as you are fond of definitions -- often, it seems, tending to read too much into them -- here is my favourite definition of outliers:

"Outliers are sample values that cause surprise in relation to the majority of the sample" (W.N. Venables and B.D. Ripley. 2002. Modern applied statistics with S. New York: Springer, p.119).

Crucially, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say if the data really are lognormal or gamma rather than normal.

P.S. I don't think that leverage points necessarily lack neighbouring observations. For example, they may occur in pairs.

• Thanks! Are outliers and high leverage points the same concept? Note that "The leverage is typically defined as the diagonal of the hat matrix" from en.wikipedia.org/wiki/Partial_leverage
– Tim
Jul 29, 2013 at 15:57
• No; you've not shown us a definition of "outlier", but it follows from the definition of leverage points that they need not be outliers sensu Venables and Ripley. (I do recommend trying to wean yourself off Wikipedia.) See also @Gael's reply. Jul 29, 2013 at 16:04
• "Crucially, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say if the data really are lognormal or gamma rather than normal." So outliers are defined wrt some model, while high leverage points and influential points aren't?
– Tim
Jul 29, 2013 at 17:00
• Venables and Ripley were, as I read it, making an intelligent point in a witty manner, and were subverting the naive idea that outliers can be defined by exact, formal statements. But other treatments can be found in different styles. In contrast, leverage and influence can be defined formally in terms of ways of measuring them. The two styles of using terminology are not really consistent. To get a better idea of what outliers are and aren't, experience of actual data analysis teaches more than reading of encyclopedia entries. Jul 29, 2013 at 17:10
• Gael referred to the comment on July 29 2013 is now using the identifier @Gala. At the time of writing there is only one other answer, but that may change. Nov 7, 2016 at 18:39

It's easy to illustrate how a high leverage point might not be influential in the case of a simple linear model:

The blue line is a regression line based on all the data, the red line ignores the point at the top right of the plot.

This point fits the definition of a high leverage point you just provided as it is far away from the rest of the data. Because of that, the regression line (the blue one) has to pass close to it. But since its position largely fits the pattern observed in the rest of the data, the other model would predict it very well (i.e. the red line already passes close to it in any case) and it is therefore not particularly influential.

Compare this to the following scatterplot:

Here, the point on the right of the plot is still a high leverage point but this time it does not really fit the pattern observed in the rest of the data. The blue line (the linear fit based on all the data) passes very close but the red line does not. Including or excluding this one point changes the parameter estimates dramatically: It has a lot of influence.

Note that the definitions you cited and the examples I just gave might seem to imply that high leverage/influential points are, in some sense, univariate “outliers” and that the fitted regression line will pass close to points with the highest influence but it need not be the case.

In this last example, the observation on the bottom right has a (relatively) large effect on the fit of the model (visible again through the difference between the red and the blue lines) but it still appears to be far away from the regression line while being undetectable in univariate distributions (represented here by the “rugs” along the axes).

• Thanks! Is the high leverage point we used here consistent with "the leverage is typically defined as the diagonal of the hat matrix" from en.wikipedia.org/wiki/Partial_leverage?
– Tim
Jul 29, 2013 at 15:57
• Excellent explanation. Would highly appreciate if you also provide the data for all three cases. Thanks Feb 23, 2016 at 1:53

Great intuitive and visual answers above, let me add a little formulas in the case of linear regression in 2 dimensions. The predictor at a given data point $$x_i$$ is $$\hat{y} = \beta_0 + \beta_1 x_i$$.

Being high leverage concern only $$x_i$$, $$|x_i- \bar{x}|$$ is unusually large compare to average. Meanwhile, being outlier concerns $$y_i$$, when the residual $$|\hat{y}-y_i|$$ given $$x_i$$ is unusually large.

For the same residual $$|\hat{y}-y|$$, a high leverage point has a stronger influence to what $$\beta_1$$ best fits the data to minimize $$\sum_i^N (\beta_0 + \beta_1 x_i-y)^2$$ because the way it forms a product with $$\beta_1$$ in the predictor. My mental picture is that a point far away from the data mean has a "longer lever" to push the slope than a point near the mean.

• This comment is the first place I have found any explanation of "why" points that are outliers in the x direction have more of an ability to exert influence on the slope than points that are aren't. Sep 26, 2022 at 3:20