6
$\begingroup$

This is more of a general statistics question, though if it matters I'm writing PHP code.

Let's say I'm trying to compute the average value of a toy that is commonly bought and sold on the secondary market, and I have a set of price values culled both from auctions and from user-entered "price paid" data. The data points that represent auctions are pretty reliable, but I also get the occasional "garage sale" type of data point, where someone may have paid a buck to buy something from Aunt Polly at a garage sale. The problem is that the $1 type of data points aren't really valuable to me, as they don't really indicate value--Aunt Polly didn't know any better, and didn't care. Similarly, I may occasionally get a data point coming from a jokester entering $9000 for a toy that is really only worth $9.

So, when computing value, what's the best way to factor these types of anomalies out of otherwise useful data?

I've read about outliers, and something about generally ignoring anything that is more than 2.5 standard deviations outside the rest of the data, but I'm looking for the full recipe, here.

Thanks so much!

$\endgroup$
  • $\begingroup$ The full recipe, is that you can control the input. You should do so. Validate responses or request numeric responses. Adding a comment field could provide some valuable qualification to prices that you can review afterwards. $\endgroup$ – Brandon Bertelsen May 16 '11 at 14:07
  • $\begingroup$ Really looking for a stastical solution, but thanks, anyway. $\endgroup$ – Max May 16 '11 at 14:14
  • 2
    $\begingroup$ FYI, outlier is a valid scientific term. $\endgroup$ – user88 May 17 '11 at 17:03
9
$\begingroup$

In boxplots, values that are more than 1.5 times the IQR (interquartile range, difference between quartile 1 and 3) away from (as in: in the direction away from the median) the quartiles are typically considered outliers.

I cannot say whether this is an appropriate measure for your data, though...

$\endgroup$
  • 4
    $\begingroup$ This is the right approach. The criterion usually is 1.5 IQRs away from the quartiles, not the median. See Tukey's EDA text, for example. (I believe he originated this rule, which he analyzes rigorously in a paper buried somewhere in his collected works.) It becomes appropriate when the data are first re-expressed in a way that makes the distribution approximately symmetric about its median. For simple ways to determine a re-expression, see EDA op. cit. $\endgroup$ – whuber May 16 '11 at 15:54
5
$\begingroup$

You could consider using a trimmed mean. This would involve discarding, say, the highest 10% of values and the lowest 10% of values, regardless of whether you consider them to be bad.

$\endgroup$
  • $\begingroup$ Trimmed means are a good idea, in keeping with the need for a robust statistic. But should the OP be comfortable with just 10% trimming? Maybe 20% trimming would be better? This thinking brings us pretty quickly to one extreme of just reporting the median (a 50% trimmed mean). The other extreme is some analysis of outliers, rejection of the outliers, and re-estimation of whatever statistics one wants. $\endgroup$ – whuber May 16 '11 at 22:05
  • $\begingroup$ I'm not exactly an expert, but just tossing out the highest and lowest 10% sounds very different from tossing out values that are clearly anomalous. If I have a hundred values that are around where I would expect them, why would I want to trim any of them? It's only the one or two out in left field I want to catch and discard... $\endgroup$ – Max May 16 '11 at 23:57
  • $\begingroup$ @whuber I just used 10% as an example. I don't know what the best choice would be, or even how "best" would be defined. @Max I don't know that trimmed means are definitely good for your situation, but at least you would be consistent and avoid subjective decisions on what to discard. $\endgroup$ – mark999 May 17 '11 at 7:04
  • $\begingroup$ @Max The idea is that removing a few values at both ends of the data is unlikely to affect your estimate of where most of the values lie, whether or not what you throw out is an outlier. I'm just pushing that point a little, knowing how bad Web data can be, and suggesting an even simpler approach of using the middle value to estimate your typical price: that's the same as "tossing out" all but a single value! You could also look into a closely related statistic, the Winsorized mean $\endgroup$ – whuber May 17 '11 at 14:05
2
$\begingroup$

I originally posted this on SO before it was deleted:

https://stats.stackexchange.com/ will probably help you better with this, and give a more comprehensive answer. I'm not a mathematician, but I suspect there are multiple ways to solve this issue.

As a programmer this is how I would tackle the problem. I'm not skilled enough to tell you if this is sound, but for simple data it should be acceptable.

Depending on the type of data, it might be acceptable to have cut off amounts. You will probably want a rolling average (often used in stock markets) that takes the average price over the last n months, this helps negate the impact of inflation, and then have a $n cuttoff or a percentage based cutoff, that is, any value that deviates +-20% or +-$n of the rolling average will be ignored.

This would work quite well for relatively stable markets, if your entity exists in a volatile market that fluctuates wildly then you probably want to find a different approach.

You also need to seriously consider cutting data off, you mention granny's yard sale which is arguably a legitimate cut off, but you need to accept that you will probably be losing legitimate data points as well that could have a significant effect on your results.

But again, there will be multiple ways to achieve this.

$\endgroup$
  • $\begingroup$ The data set I'm working with represents the most recent 6 months of data, so that much is already in place. And thanks for the help, but I'm really looking for a solution stated in statistical terms. $\endgroup$ – Max May 16 '11 at 14:17
0
$\begingroup$

Perhaps a robust estimator like RANSAC could be used here.

$\endgroup$
0
$\begingroup$

hope this helps

Simplistic approaches , as suggested here , often fail to their lack of generality. In general you may have a series that has multiple trends and/or multiple levels thus to detect anomalies one has to "control" for these effects. Additionally there may be a seasonal effect that may have started in the last k periods and not present in the first n-k values. Now let's get to the meat of the problem. Assume that there are no mean shifts/no trend changes/no seasonal pulse structure in the data. The data may be autocorrelated causing the simple standard deviation to be over or under estimated depending upon the nature of the autocorrelation. The possible existence of Pulses,Seasonal Pulses,Level Shifts and/or local time trends obfuscates the identification of the "exceptions". Using a "bad standard deviation" to try to identify anomalies is flawed because it is an out of model test as compared to an "in model test" which ultimately is what is used to conclude about the statistical significance of the anomilies. You might Google "how to do statistical intervention detection" to help you find sources/software to do this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.