2
$\begingroup$

I would like to remove 10% of my data points (I consider them as outliers) to maximize the R squared. Is there a way to do so efficiently?

I know many people suggest not to remove outliers. But in this case, I just would like the regression model to represent 90% of the population.

Improve this question
$\endgroup$
8
  • $\begingroup$ You can. I even argue that you should. Look at this answer for principled methods to do that. Note that removing outliers will not in general increase the $R^2$! $\endgroup$
    – user603
    Commented Sep 18, 2015 at 15:52
  • 1
    $\begingroup$ Instead of dropping datapoints, would you be interested in a more refined approach called robust regression in which you attach very little weight to outliers? $\endgroup$
    – JohnK
    Commented Sep 18, 2015 at 16:13
  • 3
    $\begingroup$ If you remove any proportion of the data in order to optimize some statistic, the remaining data no longer represent the population in any way that would be correctly analyzed using standard statistical procedures (that is, those which assume your data are a random sample). Specifically, all confidence intervals, prediction intervals, and p-values would be erroneous--and perhaps extremely so when as much as 10% of the data are removed. $\endgroup$
    – whuber ♦
    Commented Sep 18, 2015 at 17:10
  • 1
    $\begingroup$ @user2316040: The last sentence in my comment is very much trying to make the same point as whuber's comments. Note that the methods described at the answer I linked to are all consistent: the non-trimmed points are representative of the population from which they are drawn (in the absence of contamination, they converge to the same fit as the usual methods). $\endgroup$
    – user603
    Commented Sep 18, 2015 at 19:28
  • 1
    $\begingroup$ Given a choice, I suspect most customers would prefer "accurate" and "correct" over "easy to understand" but "potentially with strong bias or error." $\endgroup$
    – whuber ♦
    Commented Sep 21, 2015 at 14:50

2 Answers 2

Reset to default
4
$\begingroup$

You could try LTS regression: Least trimmed squares, https://en.wikipedia.org/wiki/Least_trimmed_squares. This is not first rejecting outliers, then fitting a regression, but effectively doing both at once, where outlier is defined as the points least fitting for the regression models. There is an implementation in the package mass (on CRAN) for R. There is no closed form solution for the estimators, a method resembling a genetic algorithm is used to give a close to optimal solution.

I think this is the closest fit for what you are asking for.

Note that robustness can mean many different things! and that my answer emphasizes robustness with respect to the $X$-space, that is, LTS can be useful in cases with low-quality data, where the data can contain some observations that really do not belong there, that do not correspond to the linear model you want to fit. Harrell's answer is the case with robustness with respect to $Y$-space, a different case. From your post we cannot really decide which fits your case, you must decide!

Improve this answer
$\endgroup$
4
$\begingroup$

Ordinal regression is very robust in the $Y$-space and can be used to generate interpretable estimates of quantities of interest. See for example the R rms package orm function which efficiently allows for thousands of intercepts (to handle the case where $Y$ is continuous). Note that with ordinal models you can have $N-1$ intercepts for $N$ observations (if no ties in $Y$) and this does not entail any overfitting, due to the order restrictions on these intercepts. See for example Which model should I use to fit my data ? ordinal and non-ordinal, not normal and not homoscedastic

Improve this answer
$\endgroup$
1
  • $\begingroup$ I did not mention that it is a logistic regression in this case. So they are probably X-space outliers. $\endgroup$
    – Fan
    Commented Sep 22, 2015 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.