# What are the statistical drawbacks of my naive and simple approach to discarding outliers?

I'm trying to measure and compare the performance of some computer programs. The simplest idea that comes to mind is to time, say, 10 runs in a row and present the arithmetic mean of the 10 timings as the final result. This however is quite inadequate and will produce an inflated result because individual timings will very often be significantly inflated, what I consider to be a measurement error (modern computer architectures cause highly nondeterministic performance for programs, except some embedded ones).

Example: there are three timings of the same program with the same input and output: 16.6483 seconds, 16.6626 seconds and 17.4575 seconds. The third value is obviously the result of an irreproducible and highly nondeterministic event, perhaps related to, e.g., CPU frequency scaling, CPU cache, ASLR, process scheduling, some kernel mischief, etc; and making it contribute to the mean calculation would make the result meaningless, at least with the small sample sizes that are relevant here.

I should note that some relatively big names actually advocate always taking only the minimum as the final measurement result, but I suspect that this is a quite bad choice because the minimum may represent some quite improbable alignment of events that may not be significant (if it is unlikely enough) to real program performance.

If it isn't clear by now, my goal was get a final result that would be robust to those kinds of measurement "errors", and I assume that high-valued outliers repeat fairly often.

A quick web search and skimming some Wikipedia pages indicates that this is a quite common problem and well-studied in statistics. However I'm wondering about the quality of this very simple method that should get rid of the measurement errors: for each program, take 12 measurements, sort them, delete the top 4 timings, return the mean of the 8 remaining timings

• Not on the statistical side of your question, but I guess you are comparing wall-clock times ; comparing CPU times might allow to prevent outliers : pythoncentral.io/measure-time-in-python-time-time-vs-time-clock Commented Dec 8, 2020 at 18:00
• Throwing away data will usually taste slightly bad: how can you know that the 4 longest runs are "bad"? Why 4 exactly? If you know some magic threshold of execution time that separates "true" times from "false" ones, why measure at all? If you're interested in representing what actually happens, why hide the worst runs? Commented Dec 9, 2020 at 13:01