I have a situation where I'm sampling a value (computer program runtimes, not that that should make a difference) where the sampled values are presumed to be distributed around an unknown "true" value m, with a small amount of added Gaussian noise. (The standard deviation of the noise function is unknown, but the magnitude of the standard deviation is assumed to be small relative to the absolute value of m.)
The complication is that some small amount of time the sampling process will silently "fail", drawing the reported value from a different, unknown distribution of unknown properties. The exact fraction of the time this second distribution will be sampled from is unknown, but we can place an upper bound on how frequently this happens. (In our particular case the failures are related to complex interactions with other programs on the computer, which are beyond our ability to correct or ameliorate.)
The hope is to come across some process by which the value of m can be ascertained within some set confidence interval, while minimizing the number of samples needed to determine that confidence interval. That is, we keep drawing samples until we're, for example, 95% confident that m lies within 7% of the currently calculated average, and then we stop sampling. The process needs to be automated, so plotting and manual inspection of the values is not feasible.
I'm not sure classical outlier detection schemes (e.g. those listed here) would work, as those typically presume that the sample size is fixed. The question "Correcting for outliers in a running average" is somewhat similar, but my situation is slightly different in that I wish to minimize the number of samples needed, rather than calculate a running average over a fixed window size.