15
$\begingroup$

From Wikipedia

Influential observations are those observations that have a relatively large effect on the regression model's predictions.

From Wikipedia

Leverage points are those observations, if any, made at extreme or outlying values of the independent variables such that the lack of neighboring observations means that the fitted regression model will pass close to that particular observation.

Why is the following comparison from Wikipedia

Although an influential point will typically have high leverage, a high leverage point is not necessarily an influential point.

$\endgroup$
  • 2
    $\begingroup$ The answers below are good. It may also help to read my answer here: Interpreting plot.lm(). $\endgroup$ – gung Nov 13 '14 at 18:35
12
$\begingroup$

Imagine any regression line fitted to some data.

Now imagine an extra data point, an outlier some distance away from the main body of the data, but one which lies somewhere along that regression line.

If the regression line were to be refitted, the coefficients would not change. Conversely, deleting the extra outlier would have zero influence on the coefficients.

So, an outlier or leverage point would have zero influence if it were perfectly consistent with the rest of the data and the model that rest implies.

For "line" read "plane" or "hyperplane" if desired, but the simplest example of two variables and a scatter plot is enough here.

However, as you are fond of definitions -- often, it seems, tending to read too much into them -- here is my favourite definition of outliers:

"Outliers are sample values that cause surprise in relation to the majority of the sample" (W.N. Venables and B.D. Ripley. 2002. Modern applied statistics with S. New York: Springer, p.119).

Crucially, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say if the data really are lognormal or gamma rather than normal.

P.S. I don't think that leverage points necessarily lack neighbouring observations. For example, they may occur in pairs.

$\endgroup$
  • $\begingroup$ Thanks! Are outliers and high leverage points the same concept? Note that "The leverage is typically defined as the diagonal of the hat matrix" from en.wikipedia.org/wiki/Partial_leverage $\endgroup$ – Tim Jul 29 '13 at 15:57
  • 1
    $\begingroup$ No; you've not shown us a definition of "outlier", but it follows from the definition of leverage points that they need not be outliers sensu Venables and Ripley. (I do recommend trying to wean yourself off Wikipedia.) See also @Gael's reply. $\endgroup$ – Nick Cox Jul 29 '13 at 16:04
  • 1
    $\begingroup$ "Crucially, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say if the data really are lognormal or gamma rather than normal." So outliers are defined wrt some model, while high leverage points and influential points aren't? $\endgroup$ – Tim Jul 29 '13 at 17:00
  • 1
    $\begingroup$ Venables and Ripley were, as I read it, making an intelligent point in a witty manner, and were subverting the naive idea that outliers can be defined by exact, formal statements. But other treatments can be found in different styles. In contrast, leverage and influence can be defined formally in terms of ways of measuring them. The two styles of using terminology are not really consistent. To get a better idea of what outliers are and aren't, experience of actual data analysis teaches more than reading of encyclopedia entries. $\endgroup$ – Nick Cox Jul 29 '13 at 17:10
  • $\begingroup$ Gael referred to the comment on July 29 2013 is now using the identifier @Gala. At the time of writing there is only one other answer, but that may change. $\endgroup$ – Nick Cox Nov 7 '16 at 18:39
18
$\begingroup$

It's easy to illustrate how a high leverage point might not be influential in the case of a simple linear model:

High leverage but not too influential point

The blue line is a regression line based on all the data, the red line ignores the point at the top right of the plot.

This point fits the definition of a high leverage point you just provided as it is far away from the rest of the data. Because of that, the regression line (the blue one) has to pass close to it. But since its position largely fits the pattern observed in the rest of the data, the other model would predict it very well (i.e. the red line already passes close to it in any case) and it is therefore not particularly influential.

Compare this to the following scatterplot:

High leverage highly influential point

Here, the point on the right of the plot is still a high leverage point but this time it does not really fit the pattern observed in the rest of the data. The blue line (the linear fit based on all the data) passes very close but the red line does not. Including or excluding this one point changes the parameter estimates dramatically: It has a lot of influence.

Note that the definitions you cited and the examples I just gave might seem to imply that high leverage/influential points are, in some sense, univariate “outliers” and that the fitted regression line will pass close to points with the highest influence but it need not be the case.

Hidden highly influential point

In this last example, the observation on the bottom right has a (relatively) large effect on the fit of the model (visible again through the difference between the red and the blue lines) but it still appears to be far away from the regression line while being undetectable in univariate distributions (represented here by the “rugs” along the axes).

$\endgroup$
  • $\begingroup$ Thanks! Is the high leverage point we used here consistent with "the leverage is typically defined as the diagonal of the hat matrix" from en.wikipedia.org/wiki/Partial_leverage? $\endgroup$ – Tim Jul 29 '13 at 15:57
  • $\begingroup$ Excellent explanation. Would highly appreciate if you also provide the data for all three cases. Thanks $\endgroup$ – MYaseen208 Feb 23 '16 at 1:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.