How to optimize a regression by removing 10% "worst" data points? I would like to remove 10% of my data points (I consider them as outliers) to maximize the R squared. Is there a way to do so efficiently?
I know many people suggest not to remove outliers. But in this case, I just would like the regression model to represent 90% of the population.
 A: You could try LTS regression: Least trimmed squares, https://en.wikipedia.org/wiki/Least_trimmed_squares.  This is not first rejecting outliers, then fitting a regression, but effectively doing both at once, where outlier is defined as the points least fitting for the regression models. There is an implementation in the package mass (on CRAN) for R.  There is no closed form solution for the estimators, a method resembling a genetic algorithm is used to give a close to optimal solution.
I think this is the closest fit for what you are asking for.
Note that robustness can mean many different things! and that my answer emphasizes robustness with respect to the $X$-space, that is, LTS can be useful in cases with low-quality data, where the data can contain some observations that really do not belong there, that do not correspond to the linear model you want to fit. Harrell's answer is the case with robustness with respect to $Y$-space, a different case.  From your post we cannot really decide which fits your case, you must decide!
A: Ordinal regression is very robust in the $Y$-space and can be used to generate interpretable estimates of quantities of interest.  See for example the R rms package orm function which efficiently allows for thousands of intercepts (to handle the case where $Y$ is continuous).  Note that with ordinal models you can have $N-1$ intercepts for $N$ observations (if no ties in $Y$) and this does not entail any overfitting, due to the order restrictions on these intercepts.  See for example Which model should I use to fit my data ? ordinal and non-ordinal, not normal and not homoscedastic
