The title basically says it all; what is considered to be a proper statistical test in the literature for comparing small samples of unknown distribution? That is to say, I run an experiment something like 10 times (because of the long running time, getting a significantly larger sample is not possible) under certain settings (say configuration x), and I run it another 10 times under different settings (say configuration y), and so I get two sets of 10 numbers which indicate performance, one set of 10 results for x and one set of 10 results for y. I want to know whether the differences between these samples are statistically significant, such that I can say something like "running the experiment with configuration x gives significantly higher/lower results than running it with configuration y".
Specifically, I am doing an experiment in which I run an algorithm and get a performance indicator back (a number); the question is whether the difference I see (I would generally compare them myself by average), is actually statistically significant.
From what I know Student's t-test is often considered quite good; though it assumes your distribution is like a normal distribution. I've seen Wilcoxon's signed-rank test used for cases like this, but I'm not sure whether it's really proper (I'm not an expert on statistical analysis). As far as I'm aware it also makes a similar assumption that your results are normally distributed; perhaps that's not a weird assumption if you're just re-running the same experiment (which has some random elements) a number of times, I don't know. I could say the results look like they might be normally distributed, but what do I know, especially with a sample size of 10?
