3
$\begingroup$

I have a dataset, which contains measurments from many different conditions. Since my hypothesis suggested a large difference for the measurements in each condition, in order to clean the data, I analyzed all conditions independently.

I.e. I grouped all measurments from one conditions into quartiles, calculated the interquartile range and then removed any datapoint, which was further than 1.5 times the size of the interquartile range from the median.

Now someone looked over the numbers and remarked, that the rate of removed data was much higher than usual. I re-checked my calculations several times, and arrived at such a high ratio of outliers each time.

Now I was thinking, that if I had analyzed all measurements from each condition together, most likely much less measurements would have been removed, due to the differences between the conditions. However as far as I understand these cleaning methods, they are meant to be used on single (gaußian) distributions, and not the sum of two or more distributions. So which method for data cleaning would actually be the correct one: Cleaning all measurements together or cleaning each condition separately?

Improve this question
$\endgroup$
9
  • $\begingroup$ Can you clarify what you mean by "they are meant to be used on single (gaußian) distributions"? I think when you wrote "single (gaußian) distributions" you meant "univariate (gaußian) distributions"? Is this correct? $\endgroup$
    – user603
    Commented Aug 20, 2012 at 20:37
  • $\begingroup$ Also how many conditions do you have? How many repetition (i.e. observations?) $\endgroup$
    – user603
    Commented Aug 20, 2012 at 20:38
  • $\begingroup$ @user603: No, I was not refering to univariate gaußian distributions. What I meant was, that the measurements for each conditions are distributed in shape of a gaussian. However mean and variance for all these gaussians is not the same. Thus when I take all measurements for all conditions together, the resulting distribution will be the sum of several gaussians. $\endgroup$
    – LiKao
    Commented Aug 20, 2012 at 21:14
  • 1
    $\begingroup$ I think you may have misinterpreted a common recipe for screening data: usually one looks $1.5$ times the size of the IQR range from either quartile, not from the median. After all, for a Gaussian distribution the IQR estimates $1.35$ SDs and the median estimates the mean, whence $1.5$ IQRs away from the median is only $2$ SDs from the mean, which is not rare at all: typically, between $10$ and $45$ out of $600$ such values will be that extreme. With the usual recipe, that number reduces to between $0$ and $10$. Using either rule to discard data automatically is usually a bad idea, though. $\endgroup$
    – whuber ♦
    Commented Aug 20, 2012 at 22:06
  • 1
    $\begingroup$ @LiKao: you could follow a uni-variate rule (in spirit similar to what you were doing, although using the correct rejection threshold as whuber pointed out). That would amount to assuming independence among your measurements. You could also take the correlation explicitly unto account, using a multivariate robust approach. Contrary to your intuition, analyzing the data-set as independent realization (as you seem to be doing) will lead you to understate the true contamination rate. $\endgroup$
    – user603
    Commented Aug 22, 2012 at 18:40

1 Answer 1

Reset to default
1
$\begingroup$

First of all it is my opinion and probably the opinion of many statisticians that autimatically "cleaning" (i.e. removing extreme observations) is a bad idea. Data should not be removed unless there is a reason to think there is an error. What basis do you have to think that each group of data should follow a normal distribution? It may not and the data may be perfectly valid. By doing this you are forcing the data to "look" normal. I find it particularly suspect if you are removing a lot of observations.

I have done a lot of research on outlier detection in my time. For two years while work at the Oak Ridge National Laboratory I concentrated on outlier detection in my job to validation DOE data sets. I published articles in the early 1980s in JASA and the American Statistician on outlier detection.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is a very common miss-understanding and perpetuates a somewhat caricatural view of modern robust procedures so i will address it here. The aim of modern robustness is not to delete observations that are somehow nonconforming but to limit (or bound) the influence any single observation is allowed to have on the final estimate. If you can identify a small subset of the data, that when down-weighted, completely changes the estimated results, then, it's just common sense that that subset has to be set aside (and perhaps studied separately) from the rest of the data. $\endgroup$
    – user603
    Commented Aug 21, 2012 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.