Influential observations and Outliers in Linear Regression model

What is the difference between influential observations and outliers in linear regression model?

As part of a study on airline scheduling in the 1990's, data on 100 randomly chosen nonstop flights between US cities by a major airline were collected. Here we are concerned with flight distance and scheduled flight time. These data provide examples of points that are 'influential' and points that are 'outliers' from the regression line.

A scatterplot from Minitab shows the data along with the regression line:

An influential point. The longest flight $(x \approx 4500)$ provides an influential observation. Mostly because it is an extreme observation on the x = miles scale, it heavily influences how the regression line is drawn. (This is a flight from the airline's main hub to Hawaii.) In order to minimize the sum of squared distances between data points and corresponding points on the regression line, one might say that this observation "pulls the regression line upward" at the right side of the graph. (Sometimes influential points are also outliers from the line.)

An outlier from the regression line. Another data point $(x \approx 1500)$ shows an example of a flight that is an outlier from the line, in the sense that it has an unusually large (positive) residual, but is not an influential point. (This is a flight for which extra time seems to have been allowed in the schedule because it connects especially congested airports at a busy times of day.) It turns out not to be 'influential' because there are many other flights of nearly the same distance with more typically scheduled flight times. Notice that this point is not an outlier-- either among the $x$-values (miles) or among the $y$-values (scheduled flight time). However, it is an outlier from the regression line because it lies relatively far above the line.

Notes:

(1) The slope and intercept have direct physical interpretations in this example. The slope 0.001895 hours per mile corresponds to about 528 MPH. That is about the average speed in flight of airplanes used by the carrier at that time (safely below the speed of sound at flying altitude). The intercept 0.6463 hours (about 39 min) corresponds roughly to scheduled time allowance on the ground taxiing and waiting in line for take-off.

(2) You may notice a 'fanning out' as variability about the regression line tends to increase with distance. Westbound flights tend to be faster and eastbound flight slower because of prevailing winds. By taking flight direction into account in a subsequent regression, it was possible to match the assumption that the variability of the regression model is the same throughout the range of $x$ values.

(3) Various statistical software packages have different methods of determining and indicating influential points and ones that have relatively large residuals. Minitab's printout uses X to indicate an influential point and R to indicate a point with a standardized residual larger than 2. (Obs is the sequence of the observation in the data sheet; Fit is the corresponding point of the regression line.) For the regression of the plot above here is the relevant portion of Minitab's printout:

Obs  TravTime     Fit    Resid  Std Resid
11    9.5833  9.1735   0.4099       1.72     X
28    5.2500  4.6986   0.5514       2.10  R
29    5.2833  4.6986   0.5847       2.23  R
54    7.0000  7.8151  -0.8151      -3.28  R  X
55    4.9667  5.4868  -0.5201      -2.00  R
62    3.7500  4.3330  -0.5830      -2.22  R
71    4.0833  3.2891   0.7942       3.00  R
75    6.2500  5.5493   0.7007       2.70  R


Acknowledgment: This dataset is discussed in Ch 13 of Learning Statistics with Real Data (2002) Duxbury Press. (out of print)

Addendum: In a Comment @MichaelHardy has suggested a regression with logged Miles and Travel time, and I have responded in a comment. Here is a plot of the data on log scales with a regression line based on logged data.

Adjusted $R^2$ for a subsequent regression with a second variable encoding flight direction increases to about 99%.

• Both influential observation and outliers does not fit the trend.But the former one has much influence on the estimated regression equation than the later one. Is my intuition of understanding from your answer is right? – Moni Sha Jun 10 '18 at 16:32
• If I saw this scatterplot and had all the data, the first thing I would do is look at the plot of log(travel time) against log(miles). – Michael Hardy Jun 10 '18 at 17:31
• Taking logs does indeed improve the initial regression a bit. But that is an illusory gain. It would only hide the fact that additional info (direction of travel, with or against jet stream) is needed. Once the add'l variable is added, scatter about the model is greatly reduced; only a few outliers remain (reflecting special conditions at airports served). Also, because flight conditions/restrictions are much different over the ocean btw the west coast and Honolulu than over land, one wonders whether it is reasonable to include flights to Hawaii in the sample. // See addendum for logged plot. – BruceET Jun 10 '18 at 20:54
• 100 published flight schedules: airports of departure and origin; distance[ scheduled departure, arrival, and elapsed flight time; direction codes. A bit more info. – BruceET Jun 12 '18 at 20:17
• I agree with your willingness to seek new methods (in your bio for this site), so I don't suppose you would give up seeking relevant info. However, this business of taking logs is kind of a sore point with me. Half a dozen times in my consulting I have rejected unmotivated logged variables and insisted on looking deeper thus saving a project. (Sometimes extra info, sometimes powers or products of variables already at hand.) – BruceET Jun 13 '18 at 3:45