Computing average value ignoring outliers This is more of a general statistics question, though if it matters I'm writing PHP code.
Let's say I'm trying to compute the average value of a toy that is commonly bought and sold on the secondary market, and I have a set of price values culled both from auctions and from user-entered "price paid" data. The data points that represent auctions are pretty reliable, but I also get the occasional "garage sale" type of data point, where someone may have paid a buck to buy something from Aunt Polly at a garage sale. The problem is that the $1 type of data points aren't really valuable to me, as they don't really indicate value--Aunt Polly didn't know any better, and didn't care. Similarly, I may occasionally get a data point coming from a jokester entering $9000 for a toy that is really only worth $9.
So, when computing value, what's the best way to factor these types of anomalies out of otherwise useful data?
I've read about outliers, and something about generally ignoring anything that is more than 2.5 standard deviations outside the rest of the data, but I'm looking for the full recipe, here.
Thanks so much!
 A: You could consider using a trimmed mean. This would involve discarding, say, the highest 10% of values and the lowest 10% of values, regardless of whether you consider them to be bad.
A: I originally posted this on SO before it was deleted:
https://stats.stackexchange.com/ will probably help you better with this, and give a more comprehensive answer. I'm not a mathematician, but I suspect there are multiple ways to solve this issue.
As a programmer this is how I would tackle the problem. I'm not skilled enough to tell you if this is sound, but for simple data it should be acceptable.
Depending on the type of data, it might be acceptable to have cut off amounts. You will probably want a rolling average (often used in stock markets) that takes the average price over the last n months, this helps negate the impact of inflation, and then have a $n cuttoff or a percentage based cutoff, that is, any value that deviates +-20% or +-$n of the rolling average will be ignored.
This would work quite well for relatively stable markets, if your entity exists in a volatile market that fluctuates wildly then you probably want to find a different approach.
You also need to seriously consider cutting data off, you mention granny's yard sale which is arguably a legitimate cut off, but you need to accept that you will probably be losing legitimate data points as well that could have a significant effect on your results.
But again, there will be multiple ways to achieve this.
A: In boxplots, values that are more than 1.5 times the IQR (interquartile range, difference between quartile 1 and 3) away from (as in: in the direction away from the median) the quartiles are typically considered outliers.
I cannot say whether this is an appropriate measure for your data, though...
A: Perhaps a robust estimator like RANSAC could be used here.
A: hope this helps
Simplistic approaches , as suggested here , often fail to their lack of generality. In general you may have a series that has multiple trends and/or multiple levels thus to detect anomalies one has to "control" for these effects. Additionally there may be a seasonal effect that may have started in the last k periods and not present in the first n-k values. Now let's get to the meat of the problem. Assume that there are no mean shifts/no trend changes/no seasonal pulse structure in the data. The data may be autocorrelated causing the simple standard deviation to be over or under estimated depending upon the nature of the autocorrelation. The possible existence of Pulses,Seasonal Pulses,Level Shifts and/or local time trends obfuscates the identification of the "exceptions". Using a "bad standard deviation" to try to identify anomalies is flawed because it is an out of model test as compared to an "in model test" which ultimately is what is used to conclude about the statistical significance of the anomilies. You might Google "how to do statistical intervention detection" to help you find sources/software to do this. 
