5
$\begingroup$

I'm running a linear regression in R on a dataset with 8 independent variables. When I run the model with all variables:

fullmodel <- lm(price ~ ., data = basedata)
summary(fullmodel)

I get an adj. R2 of 0.9901. However, there is a very clear outlier in the data, as confirmed by car::outlierTest(fullmodel). This observation is not a coding error but is nevertheless not representative of the wider population, so I've taken the decision to drop it. However, when I run a revised model with this observation excluded:

fullmodel_nooutlier <- lm(price ~ ., data = nooutlier)
summary(fullmodel_nooutlier)

My model diagnostics look much better using par(mfrow=c(2,2)) plot(fullmodel_nooutlier) and no assumptions of linear regression are violated so I do not believe any further data transformation is required. However my adj. R2 falls to 0.8589. I'm not sure how such an extreme outlier could reduce the model fit, so would appreciate any advice on whether or not I should retain it for analyses.

$\endgroup$
3
  • 3
    $\begingroup$ Good search terms to include with "regression" are "leverage" and "influence." For an analysis in terms of correlation coefficients, see stats.stackexchange.com/a/562760/919. stats.stackexchange.com/a/415893/919 has a graphical representation of influence and stats.stackexchange.com/a/637357/919 shows examples where one point has arbitrarily high influence: that alone can explain what you observe. $\endgroup$
    – whuber
    Commented Mar 7 at 14:24
  • 1
    $\begingroup$ That's extraordinarily helpful - I don't appear to be able to 'like' this comment but very much appreciate the insight. Thanks. $\endgroup$
    – S. Dolan
    Commented Mar 7 at 16:35
  • 3
    $\begingroup$ Often, and perhaps here, it makes more sense to work with price on logarithmic scale. That need not mean transformation; it could mean some generalized linear model. $\endgroup$
    – Nick Cox
    Commented Mar 7 at 17:16

4 Answers 4

13
$\begingroup$

Seems to be a situation like in the well known Anscombe quartet, the lower right graph with Y4. Probably your situation is less dramatic, in the sense the vertical bar of 9 points (at the left in that graph) is more stretched out horizontally so that you still have quit some R square left. Use e.g. Cook's distance to determine the fit of the outlier to the regression plane estimated without the outlier. Or similar measures (like sdresid ...).

enter image description here

Aonther possibility, like you may have:

enter image description here

The point in the upper right corner is very close to the regression line determined by only the 9 points at the left. This means that the outlying point fits very well; it has a low sdresid value (I didn't calculate it), and there is no reason to remove it from your data, as it doesn't "disturb" the regression coefficients. The 9 points at the left have an R square of 0.79, whereas including also the upper right point leads to an R square of 0.98!

$\endgroup$
3
  • 4
    $\begingroup$ I also enjoy the datasaurus dozen. $\endgroup$
    – Galen
    Commented Mar 7 at 17:46
  • $\begingroup$ @Galen Great data indeed, never heard of. $\endgroup$
    – BenP
    Commented Mar 7 at 21:18
  • $\begingroup$ Even when the outlying point doesn't fit at all with a line through the remaining points, you can still get a much higher $R^2$ with an outlier present -- if it's sufficiently far away from the main group of points. e.g. x=rnorm(19);y=-.6*x+.8*rnorm(19); x1=c(x,100);y1=c(y,100); cor(x,y);cor(x1,y1) $\endgroup$
    – Glen_b
    Commented Mar 7 at 22:01
5
$\begingroup$

A very straight answer is that $R^2$ (adjusted or not) is itself strongly affected by outliers, and that a leverage outlier (position in $x$-space far away from the majority of the data) can result in a very high value of $R^2$ that should not be trusted. So the fact that $R^2$ goes down alone should not be a reason against removing the outlier.

Note however that if the outlier is not an erroneous observation, and you remove it, you in fact remove valid information, which is not normally recommended. You could keep it and run a robust regression as recommended in other answers.

Note in particular that the statement that "no assumptions are violated" is meaningless as removal of points can make any data set look like if assumptions were fulfilled, but data dependent manipulation of the data set in fact violates assumptions itself (any data based manipulation will destroy independence), apart from the fact that model assumptions are never exactly fulfilled in reality anyway.

$\endgroup$
4
$\begingroup$

It is worth noting that if using a robust regression that is less sensitive to outliers, an R-Squared value will not be provided.

As an example, here is a linear regression in R that measures the impact of numerous independent variables on internet usage:

> summary(ols <- lm(usage ~ income + videohours + webpages + gender +
                    age, data = trainset))

Call:
lm(formula = usage ~ income + videohours + webpages + gender + 
    age, data = trainset)

Residuals:
     Min       1Q   Median       3Q      Max 
-11647.8  -2474.3   -333.5   2087.6  15830.3 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -3933.2197   527.6324  -7.454 2.44e-13 ***
income          0.8432     0.0497  16.967  < 2e-16 ***
videohours    993.7040    34.3884  28.896  < 2e-16 ***
webpages      106.3871    12.9236   8.232 7.91e-16 ***
gender        161.1798   273.0253   0.590    0.555    
age            90.6160    11.5869   7.821 1.74e-14 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3788 on 766 degrees of freedom
Multiple R-squared:  0.8257,    Adjusted R-squared:  0.8245 
F-statistic: 725.6 on 5 and 766 DF,  p-value: < 2.2e-16

We see that most variables are significant, and an R-Squared value of 0.8257 is provided.

However, when looking at the QQ plot, we see that outliers are clearly present:

qq plot

Given that the regression results will be impacted by those outliers, one solution is to use a special type of weighted regression that appends less weight to outliers. For this purpose, let us use a Huber regression:

> rr.huber <- rlm(usage ~ income + videohours + webpages + gender + 
                  age, data = trainset)
> summary(rr.huber)

Call: rlm(formula = usage ~ income + videohours + webpages + gender + 
    age, data = trainset)
Residuals:
      Min        1Q    Median        3Q       Max 
-11771.91  -2229.33    -92.45   2277.87  15964.66 

Coefficients:
            Value      Std. Error t value   
(Intercept) -4187.9110   482.9303    -8.6719
income          0.9031     0.0455    19.8536
videohours    990.4145    31.4750    31.4667
webpages      102.5059    11.8287     8.6659
gender        -21.0219   249.8941    -0.0841
age            88.7019    10.6052     8.3640

Residual standard error: 3326 on 766 degrees of freedom

In this instance, we see that an R-Squared value is not provided. The reason is that R-Squared is measuring the goodness of fit, or how well the regression line approximates the data points. However, if we are re-weighting so that the regression becomes less sensitive to outliers, then using an R-Squared measurement no longer makes sense.

Moreover, we can see that as the residuals decrease in size, weight also increases - meaning the results are appending more weight to observations closer to the regression line:

> huber <- data.frame(usage = trainset$usage, resid = rr.huber$resid, 
                      weight = rr.huber$w)
> huber2 <- huber[order(rr.huber$w), ]
> huber2[1:10, ]
    usage    resid    weight
227 24433 15964.66 0.2802074
419 17843 15697.99 0.2849742
93  27213 15362.05 0.2911959
707 29626 14114.49 0.3169422
568 25973 13995.47 0.3196348
581 23903 13281.23 0.3368172
513 25182 13067.13 0.3423320
105 19606 13015.32 0.3437127
283 22488 12862.41 0.3477962
41  15044 12193.07 0.3668919

From this standpoint, using a robust regression could be a suitable alternative method if outliers are skewing the regression results without having to modify the original set of data.

I would also recommend reading this post, which also explains why R-Squared values don't make sense when correcting for outliers in a regression model.

$\endgroup$
2
$\begingroup$

Why should the $R^2$ decrease?

$R^2$ is (or can be) calculated by comparing the total variance in $y$ to the residual variance. When you remove the outlier, you change the total variance. Sure, that might lead to a change in the residual variance, but there is also a change in the total variance.

The basic structure of the $R^2$ equation is $ R^2 = 1-\dfrac{A}{B}.$ You affect $A$ (residual variance) by removing the outlier, sure, but you also affect $B$ (total variance).

The adjusted $R^2$ tweaks the usual $R^2$ by considering the parameter count, but the argument is basically the same: removing the outlier impacts both the numerator and denominator, meaning that the fraction could increase or decrease (perhaps even stay the same).

Here is a simulation of an example.

set.seed(2024)
N <- 99
x <- runif(N, 0, 10)
Ey <- x
e <- rnorm(N, 0, 1)
y <- Ey + e

# Now introduce an outlier of sorts
x0 <- max(x) + 100
y0 <- x0 + 25
x_full <- c(x, x0)
y_full <- c(y, y0)

# R^2 without the outlier (I get 0.8728675)
#
cor(x, y)^2

# R^2 with the outlier (I get 0.952949)
#
cor(x_full, y_full)^2
$\endgroup$
1
  • 1
    $\begingroup$ This reasoning is incorrect on all points: the adjusted $R^2$ does change predictably; that change can be quantified by some measure of influence; removing an "outlier" does not necessarily lower "total variance" (whatever that might be) because it depends on the nature of the outlier; and using an adjusted $R^2$ is irrelevant because the number of variables is not changing. $\endgroup$
    – whuber
    Commented Mar 7 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.