What is the difference between influential observations and outliers in linear regression model?
As part of a study on airline scheduling in the 1990's, data on 100 randomly chosen nonstop flights between US cities by a major airline were collected. Here we are concerned with flight distance and scheduled flight time. These data provide examples of points that are 'influential' and points that are 'outliers' from the regression line.
A scatterplot from Minitab shows the data along with the regression line:
An influential point. The longest flight $(x \approx 4500)$ provides an influential observation. Mostly because it is an extreme observation on the x = miles scale, it heavily influences how the regression line is drawn. (This is a flight from the airline's main hub to Hawaii.) In order to minimize the sum of squared distances between data points and corresponding points on the regression line, one might say that this observation "pulls the regression line upward" at the right side of the graph. (Sometimes influential points are also outliers from the line.)
An outlier from the regression line. Another data point $(x \approx 1500)$ shows an example of a flight that is an outlier from the line, in the sense that it has an unusually large (positive) residual, but is not an influential point. (This is a flight for which extra time seems to have been allowed in the schedule because it connects especially congested airports at a busy times of day.) It turns out not to be 'influential' because there are many other flights of nearly the same distance with more typically scheduled flight times. Notice that this point is not an outlier-- either among the $x$-values (miles) or among the $y$-values (scheduled flight time). However, it is an outlier from the regression line because it lies relatively far above the line.
(1) The slope and intercept have direct physical interpretations in this example. The slope 0.001895 hours per mile corresponds to about 528 MPH. That is about the average speed in flight of airplanes used by the carrier at that time (safely below the speed of sound at flying altitude). The intercept 0.6463 hours (about 39 min) corresponds roughly to scheduled time allowance on the ground taxiing and waiting in line for take-off.
(2) You may notice a 'fanning out' as variability about the regression line tends to increase with distance. Westbound flights tend to be faster and eastbound flight slower because of prevailing winds. By taking flight direction into account in a subsequent regression, it was possible to match the assumption that the variability of the regression model is the same throughout the range of $x$ values.
(3) Various statistical software packages have different methods of determining and
indicating influential points and ones that have relatively large residuals.
Minitab's printout uses
X to indicate an influential point and
indicate a point with a standardized residual larger than 2. (
Obs is the
sequence of the observation in the data sheet;
Fit is the corresponding point of the regression line.) For the
regression of the plot above here is the relevant portion of Minitab's
Obs TravTime Fit Resid Std Resid 11 9.5833 9.1735 0.4099 1.72 X 28 5.2500 4.6986 0.5514 2.10 R 29 5.2833 4.6986 0.5847 2.23 R 54 7.0000 7.8151 -0.8151 -3.28 R X 55 4.9667 5.4868 -0.5201 -2.00 R 62 3.7500 4.3330 -0.5830 -2.22 R 71 4.0833 3.2891 0.7942 3.00 R 75 6.2500 5.5493 0.7007 2.70 R
Acknowledgment: This dataset is discussed in Ch 13 of Learning Statistics with Real Data (2002) Duxbury Press. (out of print)
Addendum: In a Comment @MichaelHardy has suggested a regression with logged Miles and Travel time, and I have responded in a comment. Here is a plot of the data on log scales with a regression line based on logged data.
Adjusted $R^2$ for a subsequent regression with a second variable encoding flight direction increases to about 99%.