The underlying distribution is not known explicitly but rather inferred from a sample of around 10 million cases. It represents the cost of something and after normalizing the mean to 1 it has the following properties. By definition it is always greater equal zero. There is an around 20% probability for a value of exactly zero. Looking at histograms the greater zero values are strictly increasing until around 1.2, have a single local maximum there and are then decreasing like a power law with the highest values around 100 (and a probability of around 1 in 1 million to get a value above 100). I computed the standard deviation to be around 1.5.

I have a collection of around 100000 samples from this distribution (assumed to be independent and random) of sizes ranging from a few hundred to a few hundred thousand and I want to test their sample means to find instances that are so unlikely that there is likely an error in the data of this sample.

The expected sample mean is of course always 1 regardless of size. I computed the expected standard deviation of the sample mean as 1.5/sqrt(sample size). This allows me to order the different samples by the number of standard deviations their sample mean is away from 1 and thus filter out the most unusual samples.

This works reasonably well but I'm somewhat uncomfortable with it because this filter is symmetric about deviations above and below 1 whereas the underlying distribution is clearly not. For a given sample size this gives the same probability for a sample mean < 0 as for a sample mean > 2 although I know a priori that a sample mean can't be smaller than zero whereas a sample mean above 2 can happen with positive probability.

Is there a better way to test the sample means that accounts for the skewness of the distribution?

  • $\begingroup$ Why do you want to filter out the most unusual samples? From what you describe, you are observing what you would expect to happen, and the more extreme resampled sample means are not outliers if you want outliers to mean data which are clearly not from the underlying population. $\endgroup$
    – Henry
    Aug 13 at 11:35
  • $\begingroup$ @Henry The samples are the providers of the data and the task is to check whether there are errors in the data collection. If a sample of size 1000 has one case over 100 that is probably okay. If it has 50 something went wrong somewhere because this would be astronomically unlikely with the given distribution. Similarly if it has 25% zero entries that is a bit high but probably okay. If it has 75% it looks like something went wrong somewhere. I'm trying to do this in an automated way because I have 100000 samples to check. $\endgroup$
    – quarague
    Aug 13 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.