I have a question about labeling outliers in my data. My data is weights for several products. Of course, there is a 'true weight' of a product. I wish to find/approach that from my data (on average). Due to a lot of reasons, the measured weight can be (very) different from the real weight. A simple example is that I have this vector of observations: 1,1,1,1,1,1,1,1,1,1,10. I would identify the 10 as an outlier, and because I have many observations of 1, I can assume that the real weight will be close to 1. However, maybe I have 1,1,1,1,4,4,4,5,5,5,6,6,6,8,8,8,8,8. There are no extreme outliers in this vector, but I can also not accept the mean of this vector as the 'real weight' because there is just way too much variance in the observations.

I am wondering if there is any specific method or function that can identify 'acceptable observations'. In the first case, only the 1s are acceptable. In the latter example, none of the observation is (yet) acceptable (depending on future observations).

I'm kind of lost in how to approach such a problem. I already tried the standard methods of detecting outliers, such as boxplot and 2 standard deviations. But that won't cover my last example. I also looked at the ratio of standard deviation and mean. but I am not sure how to apply that. I found that there are also different results for high absolute values and low absolute values. Any ideas?

  • 1
    $\begingroup$ Have you tried qpois(c(0.025,0.975),mean(c(1,1,1,1,1,1,1,1,1,1,10))) to get the 95 % boundaries? $\endgroup$ – user2974951 Sep 11 '18 at 10:10
  • $\begingroup$ You seem to have at least two criteria for objectionable observations. Outliers in the first example, and high variability (or possibly too many large values?) in the second. // Either a boxplot or the method suggested by @user2974951 will highlight observation 10 as unusual in your first example. I'm not sure what criterion to try in your second example (neither boxplot nor attempted Poisson fit works there). $\endgroup$ – BruceET Sep 11 '18 at 22:01
  • $\begingroup$ Maybe you know how to set bounds on how large observations can be (in which case throw out those known to be impossibly high?) or on maximum variability (throw out sample?). But be careful: The standard deviation in your first example is about 2.7 (with no complaint about high variability as such), but in the second example the SD is about 2.6. // If you can formulate a criterion for unacceptable values or samples, you might try using that in a Bayesian framework. $\endgroup$ – BruceET Sep 11 '18 at 22:07
  • $\begingroup$ Thanks for the reactions. @BruceET but it doesn't matter how large the observations are, it is more about the spread of the observations. So let's say the coefficient of variation maybe.. But then again as you say, the first one has sd 2.7 and the second one 2.6. I'm now trying some kind of combination of COV and standard outlier detection => if spread is too wide or there are outliers, I have to do something. But it is not completely what I'm looking for. Probably needs some finetuning $\endgroup$ – pk_22 Sep 13 '18 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.