How to optimize a regression by removing 10% "worst" data points?

I would like to remove 10% of my data points (I consider them as outliers) to maximize the R squared. Is there a way to do so efficiently?

I know many people suggest not to remove outliers. But in this case, I just would like the regression model to represent 90% of the population.

• You can. I even argue that you should. Look at this answer for principled methods to do that. Note that removing outliers will not in general increase the $R^2$! Sep 18 '15 at 15:52
• Instead of dropping datapoints, would you be interested in a more refined approach called robust regression in which you attach very little weight to outliers? Sep 18 '15 at 16:13
• If you remove any proportion of the data in order to optimize some statistic, the remaining data no longer represent the population in any way that would be correctly analyzed using standard statistical procedures (that is, those which assume your data are a random sample). Specifically, all confidence intervals, prediction intervals, and p-values would be erroneous--and perhaps extremely so when as much as 10% of the data are removed.
– whuber
Sep 18 '15 at 17:10
• @user2316040: The last sentence in my comment is very much trying to make the same point as whuber's comments. Note that the methods described at the answer I linked to are all consistent: the non-trimmed points are representative of the population from which they are drawn (in the absence of contamination, they converge to the same fit as the usual methods). Sep 18 '15 at 19:28
• Given a choice, I suspect most customers would prefer "accurate" and "correct" over "easy to understand" but "potentially with strong bias or error."
– whuber
Sep 21 '15 at 14:50

Note that robustness can mean many different things! and that my answer emphasizes robustness with respect to the $X$-space, that is, LTS can be useful in cases with low-quality data, where the data can contain some observations that really do not belong there, that do not correspond to the linear model you want to fit. Harrell's answer is the case with robustness with respect to $Y$-space, a different case. From your post we cannot really decide which fits your case, you must decide!
Ordinal regression is very robust in the $Y$-space and can be used to generate interpretable estimates of quantities of interest. See for example the R rms package orm function which efficiently allows for thousands of intercepts (to handle the case where $Y$ is continuous). Note that with ordinal models you can have $N-1$ intercepts for $N$ observations (if no ties in $Y$) and this does not entail any overfitting, due to the order restrictions on these intercepts. See for example Which model should I use to fit my data ? ordinal and non-ordinal, not normal and not homoscedastic