2
$\begingroup$

I have two datasets consisting of metrics from several experiments. Dataset 1 is the collection of results of experiments E performed by user A on product A, repeated N times. Dataset 2 is the collection of results of the same experiments E performed by the same user A on product B, repeated the same N times.

N is not large, and cannot be large due to practical limitations (typically around 15-20). The data CANNOT be assumed to be Gaussian. In some cases, it is known to be definitely not normal, and in some cases we do not know for sure. It is just an unknown distribution. We know that the metrics cannot be negative. That is pretty much the only definite information we know.

Using this data, how do we compare product A and product B and give a result with some statistical significance? How do we devise a hypothesis test to check if A > B with some x% confidence? Edit

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

You could do a permutation/randomization test.

Possibly Wilcoxon rank sum will answer your question as well, although permutation test is probably closer to what you want.

In R there is perm.test in the exactRankTests package that seems made for your problem.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Very good suggestion, but both Wilcoxon rank sum and the permutation test are non parametric tests whose significance tests are based on asymptotic theory and hence have lesser power for small sample sizes. That is a major concern. $\endgroup$
    – Giridhar Murali
    May 7, 2013 at 21:15
  • 1
    $\begingroup$ Randomization/permutation tests are almost designed for small samples. As far as I know, it doesn't depend on any theory at all. If samples are very small, you can list all possible permutations and have exact p value. For small samples there is permtest in the BHH2 package. $\endgroup$
    – Peter Flom
    May 7, 2013 at 21:19
  • $\begingroup$ Non-parametric tests in general are the way to go when sample sizes are small, but is it not true that they lack statistical power? How can I reject a null hypothesis with some probability using a non-parametric test? $\endgroup$
    – Giridhar Murali
    May 7, 2013 at 21:44
  • 1
    $\begingroup$ When the assumptions of a parametric method are met, non-parametric methods often have lower power (although the difference can be small). When the assumptions are not met, the parametric tests can't be used properly. In another sense, though, permutation tests have the perfect power - they are a pure distillation of what "statistical significance" means. However, my view is that effect size is more important than stat sig. $\endgroup$
    – Peter Flom
    May 7, 2013 at 21:53
  • $\begingroup$ Makes sense! I accepted your answer. One more related question: Is there a reason to prefer the permutation test over other standard non-parametric tests like the Mann Whitney U test or the Kruskal-Wallis' one way ANOVA? $\endgroup$
    – Giridhar Murali
    May 7, 2013 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.