How can I identify and remove outliers in R I am performing regression analysis on prices of product that we have purchased, based on size and other attributes.
However there are often buys in odd circumstances which factor into the price, that is not (and cannot be) addressed directly in the features of the analysis.
Each time I run a regression, I will check the 20 with the largest error manually, and 90%+ of the time they will be odd buys like mentioned before, and for my purposes can be completely ignored.
I have been looking into cooks distance to remove these, however I'm not sure how to best set the threshold, or if there is a better method to use.
 A: 
However there are often buys in odd circumstances which factor into the price, that is not (and cannot be) addressed directly in the features of the analysis.



*

*Isn't that what error terms in a regression are supposed to capture: variation in the outcome variable that isn't explained by the features of your model?


If your question is how to deal with outliers in general under the assumption that extreme observations are probably bad data
Some standard approaches are:


*

*Trimming the data. Eg. ignore 1% of most extreme observations.

*Winsorizing the data. Replace observations above or below some cutoff with the value of the cutoff. (This isn't quite extreme as trimming the data, which deletes extreme observations entirely.)


Some fancier approaches to outliers (ignore if this is at all confusing):


*

*You can do things like ellipsoidal peeling. Find the minimum volume ellipsoid which encloses your data than remove observations along the surface.

*Estimate regression with Huber Loss function or something less sensitive to outliers than OLS. Or maybe maximum likelihood estimator with t distributed rather than normal distributed errors, etc...

*Quantile regression.

*You could adopt some Bayesian view as to whether an observation is bad data.


Beware the problems of mishandling outliers...
In many cases, such as returns for financial securities, removing or ignoring outliers can be hugely problematic. Often times, all the action is in the outliers! Major stock market crashes, company bankruptcies, etc... are hugely important.
For situations involving safety, (eg. auto-crashes etc...), ignoring bad outliers can be even worse! You don't want to winsorize observations such that observations where people die get replaced with observations where people are mildly injured. That would be perhaps criminal negligence.
A: I suggest you to follow the steps described in Zuur et al (2010) A protocol for data exploration to avoid common statistical problems. This will help you identify outliers.
A: It sounds from your description that what you are doing is (a) using some automatic process to identify potential outliers (b) examining them one by one using your subject matter knowledge and eliminating those which seem to come from some other process. This seems a sensible procedure but the crucial step is step (b) and so whatever procedure you use in step (a) you really cannot give up step (b) otherwise you are letting the machine pick your model for you when you have information you can use yourself. I realise that this is not going to be much help to you if you have thousands of potential outliers to screen but human intelligence is the way to go here.
A: Another alternative to not removing the outliers would be to use ridge regression.
Ridge regression penalizes outliers reducing their influence when optimizing the regression coefficients. It does have a training parameter lambda that must be tuned using a method like cross-validation to get acceptable results.
A: A common way to remove outliers is the peel-off method (which I learnt from a friend) and which goes like this: you take your set of data points, and construct a convex hull; then you remove the boundary points from your set, and consider constructing the subsequent convex hull ;  and then you find how much shrinkage you actually performed in this process of removing data points. 
Then, based on your purpose you set some shrinkage threshold and continue this peel-off method, until you reach the shrinkage threshold, and upon reaching you stop.
This is quite easy to do in R, since you have several algorithms available for constructing convex hulls and identifying points on the boundary. 
