2
$\begingroup$

I would like to identify outliers in a sample of 6,000 observations.

My variable doesn't follow a normal distribution and my series is spread out on the right. So an extreme value is not necessary an outliers... that's my problem.

I have not found any solution to identify outliers.

I saw that there are the test of Dixon, but it's restricted to normally distributed samples which contains less than 30 observations.

$\endgroup$
  • 2
    $\begingroup$ What makes you think that there are any outliers (in the sense of erroneous extreme values)? $\endgroup$ – Henry Sep 6 '17 at 14:57
  • 2
    $\begingroup$ My study deals with the selling price of products. For example, I found cakes for 8 people under 13 cents or baby underwear 2977,68 euros. $\endgroup$ – Clarisse Dussauge Sep 6 '17 at 15:08
  • $\begingroup$ Such information should really be included in the original post! It seems maybe you need data cleaning based on domain knowledge, not outlier rejection per se. $\endgroup$ – kjetil b halvorsen Feb 9 at 13:53
  • $\begingroup$ @Henry being an outlier doesn't actually require that the data are erroneous. The definition isn't well defined. But I think that most believe it's high leverage and high influence observations $\endgroup$ – AdamO Jun 17 at 15:16
1
$\begingroup$

A statistical test has to do with replications of the experiment and a null hypothesis that is not "discovered" by the incidental finding of a outlying data point. For that reason, it doesn't make any sense to use a statistical test for data points, but you can use critical values or other criteria to flag observations as possible outliers, and then proceed accordingly to verify the data's accuracy.

Because of Chebyshev's inequality, you can always probabilistically quantify the distance of an observation from the mean in terms of a Z-score. The famous rule of Tukey identifies outliers based on a lower bound of normal of Q1 - 1.5 IQR and Q3 + 1.5 IQR. To give you a sense, in a normal distribution the upper bound comes out to a value of 2.70, which in a sample of 6,000 would flag about 21 observations irrespective of their actually being outliers.

Along those lines, it is fair to consider any rule that suits the problem to rank and classify outlying observations. Some ad hoc examples below:

  1. Use Tukey's test to flag outliers. With 6,000 you may set a FDR by simulation or something similar to scale the IQR by an even larger value as needed.
  2. Log transform the data if the data are concentrations or counts (due to biologic interest).
  3. Use a Box Cox transform to generate the optimally normal exponential change-of-variable and then apply normal tests.
  4. Use the Z-score to rank and flag outliers and choose a stringent alpha level critical value to flag outliers anyway.
  5. Use a known distribution suspected to form a data generating process, fit a QQ-plot to those data, and rank outliers in terms of mean-squared error from the calibration line.
  6. Use single-observation deletion and perform maximum likelihood to find which observation's deletion leads to the greatest improvement in likelihood.
$\endgroup$
0
$\begingroup$

As a simple approach, I like to set bounds on typical data consistent with outlier identification in usual boxplot construction, such as

lowerprice <- quantile(price, .25) - 1.5 * IQR(price)
upperprice <- quantile(price, .75) + 1.5 * IQR(price)

where price would be a vector of prices, lowerprice would be the lower bound and upperprice would be the upper bound. NOTE that I would apply this for each separate group of like products. What would be an outlier for clothes would not be an outlier for cars!

$\endgroup$
  • $\begingroup$ I'd expect this to work better with log of price. $\endgroup$ – Nick Cox Sep 7 '17 at 7:09
  • $\begingroup$ Good idea -- probably would help with a highly skewed distribution $\endgroup$ – user176334 Sep 7 '17 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.