0
$\begingroup$

The title basically says it all; what is considered to be a proper statistical test in the literature for comparing small samples of unknown distribution? That is to say, I run an experiment something like 10 times (because of the long running time, getting a significantly larger sample is not possible) under certain settings (say configuration x), and I run it another 10 times under different settings (say configuration y), and so I get two sets of 10 numbers which indicate performance, one set of 10 results for x and one set of 10 results for y. I want to know whether the differences between these samples are statistically significant, such that I can say something like "running the experiment with configuration x gives significantly higher/lower results than running it with configuration y".

Specifically, I am doing an experiment in which I run an algorithm and get a performance indicator back (a number); the question is whether the difference I see (I would generally compare them myself by average), is actually statistically significant.

From what I know Student's t-test is often considered quite good; though it assumes your distribution is like a normal distribution. I've seen Wilcoxon's signed-rank test used for cases like this, but I'm not sure whether it's really proper (I'm not an expert on statistical analysis). As far as I'm aware it also makes a similar assumption that your results are normally distributed; perhaps that's not a weird assumption if you're just re-running the same experiment (which has some random elements) a number of times, I don't know. I could say the results look like they might be normally distributed, but what do I know, especially with a sample size of 10?

$\endgroup$
  • $\begingroup$ I'm unclear on what is being compared. Are you trying to compare the 10 results to one another? All of them to some baseline expected vale? Some other comparison? $\endgroup$ – CFD Feb 19 at 19:15
  • $\begingroup$ @CFD I've modified the first paragraph, I hope this is better $\endgroup$ – Lara Feb 19 at 19:45
0
$\begingroup$

If you can't make any assumptions about the underlying distributions, you will need a non-parametric test. In this case you could use the Mann-Whitney U test (aka Wilcoxon rank-sum test). This is different from the Wilcoxon signed rank-sum test, which is used for paired data.

If you find after collecting the data that the results are approximately normal in each group (and have similar variance), you can use the t-test which will be more powerful than the Wilcoxon rank-sum test.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.