# Do I need to remove an outlier in x-space for linear regression?

I have the following dataset:

structure(list(x = c(108.448243992603, 88.4197989907866, 515.362096476673,
81.7591204447437, 99.0006958346383, 69.0624013486279, 130.845344345704,
109.747845137572, 78.1740107838889, 100.443473869653, 67.3086590944859,
70.5990381250307), y = c(3.777, 4.878, 4.503, 4.559, 4.3505,
3.13, 4.5835, 4.2985, 4.4895, 3.332, 4.5805, 6.6235)), .Names = c("x",
"y"), row.names = c(NA, -12L), class = "data.frame", na.action = structure(c(1L,
2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 11L, 13L, 14L, 15L, 18L,
19L, 29L), .Names = c("1", "2", "3", "4", "5", "6", "7", "13",
"14", "31", "32", "34", "35", "36", "39", "40", "50"), class = "omit"))


As you can see, I have a single x value that far exceeds all the others. If I remove it, then there is still no obvious relationship between x and y.

Therefore, ought I to remove this single x value before conducting a linear regression?

Full dataset:

              Estimate Std. Error t value Pr(>|t|)
(Intercept)  4.432e+00  3.875e-01  11.436 4.59e-07 ***
x           -5.159e-05  2.233e-03  -0.023    0.982


With single x value removed:

              Estimate Std. Error t value Pr(>|t|)
(Intercept)  5.34221    1.37170   3.895  0.00365 **
x           -0.01012    0.01471  -0.688  0.50852


Obviously, it makes a big difference on the regression. In this case, I'm actually hoping to minimize the confidence interval of the slope and therefore prefer the full dataset but am not sure if it's valid statistically. As I was gathering these data I intentionally took this measurement to see if a really high x measurement would cause the y value to change.

• What does it mean to hope to minimize the confidence interval of the parameter? You should hope to have a correct confidence interval for the parameter, not a small one. Aug 23, 2017 at 4:14
• That's fair. That's really what I'm looking for. Aug 23, 2017 at 4:20
• Did you try robust regression techniques ?
– user83346
Aug 23, 2017 at 6:50
• @fcop Most of the robust regression techniques I know of would fail on this example. Which one(s) would you have in mind, then? CephBirk: if you were to analyze these data in a way that follows your intentions, then you would perform the regression for all but the last value, you would construct a prediction interval for the response at the last value, and check whether the observed response falls within that interval. (Obviously it will.) Your conclusions would be (1) you have estimated regression coefficients for $67\lt x\lt131$ and (2) checked that extrapolating to $x=515$ might work.
– whuber
Aug 24, 2017 at 13:07
• @whuber: e.g. robust regression with redescending weight kernel
– user83346
Aug 25, 2017 at 6:21

Three comments that go a little beyond helpful answers to date. Although this question is posed with a particular dataset, more general issues are clearly worth discussion.

1. Obvious but important: the dataset is too small to be confident what is best to do.

Easy to say, but getting more data if you can is the best thing you can do here.

2. Should you exclude the outlier?

Excluding the outlier and not telling anyone would be poorest practice.

Excluding the outlier as anomalous or awkward is treated by others as a defensible practice. I would urge rather two regressions reported concisely, that with and that without the outlier. Sometimes the outlier makes less difference than you fear; sometimes it becomes clear that quite different models ensue. If you make a judgement that one model is better, you're still allowing readers to think about other analyses. (Often readers care little about your particular data, for example, but are gathering ideas on what analyses make sense for their similar data.)

In your case, you did precisely that. And we find that the slope appears to be about zero in both cases, so the inference is similar, that there is no linear relationship worth scientific attention.

Making your data public if possible also encourages alternative analyses, just as you are doing here.

3. Why not transform the predictor?

An option that can make sense is to transform the predictor, which will dampen the effects of any outlier(s). Here as all values are positive logarithms are in order mathematically.

Your own stance of including the observation with the highest value is equivalent to admitting that values could be much (even an order of magnitude?) larger, so what would happen then? Whenever that is the attitude, log scale is defensible.

Naturally, there is always a scientific or practical question of whether $y$ as a function of log $x$ makes sense otherwise, given known limiting behaviour (what happens at very small or very large values). Here as the variables are completely anonymous, no advice can be given on that, but your own subject-matter expertise should come into play.

Such an observation is called a high leverage point. As such it has a huge potential to influence your parameter estimates and everything related to it. How large the influence finally is also depends on the size of the residual. Ideally such observations are already identified in the preliminary descriptive analysis.

There is no general rule how to deal with such values. Options include

• Use a regression method that can better deal with outliers in x, like e.g. robust regression with redescending weight kernel

• Correct the value if it is a data error

• If you are allowed to, remove the value and clearly state that your results only apply to a certain range of x values.

• I think it is useful to note that there is a difference between an outlier in the x-direction (high-leverage point) and an outlier in your regression of y on x. A high leverage point can still follow the trend set by the other points, in which case it is not an outlier in your regression, i.e. the distribution of y conditional on x. Aug 24, 2017 at 12:29

Do you have reason to believe that it is in fact an outlier? You say that you intentionally measured it as such to check if there was an apparent change in the response - which there does not seem to be.

In general, I oppose ever removing outliers if you cannot directly explain that it in fact must be an outlier (e.g. an age of 200 years, a negative pulse or what have you). The data that you have observed is your basis for drawing inference. Can you reasonably remove any part of this basis and still believe that you are getting the full picture?

However, in cases such as this there is an obvious problem with the range of data. You have a huge gap between the left cluster and the right-most point, so any inference drawn in this range is pure guess-work. This makes, as Michael M says, the right point a high-leverage point, pretty much forcing any regression line directly through this point. And for that reason I'd probably exclude this point and simply state the range that you believe your analysis to be reasonable in.

Ideally though you should repeat your experiment but take more measurements and distribute them more evenly (so to avoid the above case).

That's my two cents at least.