# Is it ok to clean up data from different conditions independently?

I have a dataset, which contains measurments from many different conditions. Since my hypothesis suggested a large difference for the measurements in each condition, in order to clean the data, I analyzed all conditions independently.

I.e. I grouped all measurments from one conditions into quartiles, calculated the interquartile range and then removed any datapoint, which was further than 1.5 times the size of the interquartile range from the median.

Now someone looked over the numbers and remarked, that the rate of removed data was much higher than usual. I re-checked my calculations several times, and arrived at such a high ratio of outliers each time.

Now I was thinking, that if I had analyzed all measurements from each condition together, most likely much less measurements would have been removed, due to the differences between the conditions. However as far as I understand these cleaning methods, they are meant to be used on single (gaußian) distributions, and not the sum of two or more distributions. So which method for data cleaning would actually be the correct one: Cleaning all measurements together or cleaning each condition separately?

• Can you clarify what you mean by "they are meant to be used on single (gaußian) distributions"? I think when you wrote "single (gaußian) distributions" you meant "univariate (gaußian) distributions"? Is this correct? Aug 20, 2012 at 20:37
• Also how many conditions do you have? How many repetition (i.e. observations?) Aug 20, 2012 at 20:38
• @user603: No, I was not refering to univariate gaußian distributions. What I meant was, that the measurements for each conditions are distributed in shape of a gaussian. However mean and variance for all these gaussians is not the same. Thus when I take all measurements for all conditions together, the resulting distribution will be the sum of several gaussians. Aug 20, 2012 at 21:14
• I think you may have misinterpreted a common recipe for screening data: usually one looks $1.5$ times the size of the IQR range from either quartile, not from the median. After all, for a Gaussian distribution the IQR estimates $1.35$ SDs and the median estimates the mean, whence $1.5$ IQRs away from the median is only $2$ SDs from the mean, which is not rare at all: typically, between $10$ and $45$ out of $600$ such values will be that extreme. With the usual recipe, that number reduces to between $0$ and $10$. Using either rule to discard data automatically is usually a bad idea, though.
– whuber
Aug 20, 2012 at 22:06
• @LiKao: you could follow a uni-variate rule (in spirit similar to what you were doing, although using the correct rejection threshold as whuber pointed out). That would amount to assuming independence among your measurements. You could also take the correlation explicitly unto account, using a multivariate robust approach. Contrary to your intuition, analyzing the data-set as independent realization (as you seem to be doing) will lead you to understate the true contamination rate. Aug 22, 2012 at 18:40