5
$\begingroup$

We performed an experiment (usability testing), which had two independent samples, one group with 5 participants and the other with 24. Based on the analysis of the distribution of the data (Shapiro-Wilk test), we concluded that we operate with data where some measurements distributed normally and others do not confirm to normal distribution.

I was wondering which test would be the most appropriate for data that is normally distributed and which for the data that is not normally distributed in our case, since we compare a group with 5 participants with a group that has 24?

We found T-Test (for normally distributed data) and Mann Whitney test (for data that is not normally distributed), however, we have two groups that are uneven and based on the conflicting information we are not sure if we can perform these tests.

Thank you for your answer!

Improve this question
$\endgroup$
5
  • $\begingroup$ You can start with this answer: stats.stackexchange.com/questions/121852/… as it discusses sample-size issues and robustness of t-tests. $\endgroup$
    – Tim ♦
    Mar 11, 2016 at 12:19
  • $\begingroup$ The issue is not the fact that the sample sizes are uneven, but that the sample sizes are small. See the link in Tim's comment for a good discussion of the small sample size testing situation. $\endgroup$
    – user44764
    Mar 12, 2016 at 4:19
  • $\begingroup$ The conclusion that one of your samples is actually drawn from a normal distribution is not a correct conclusion from failure to reject the null in a Shapiro-Willk. Failure to detect non-normality doesn't tell you it's normal, it tells you you failed to detect the non-normality you have. What your testing shows is that at least some of your data aren't drawn from a normal distribution (or that you made a type I error, perhaps). $\endgroup$
    – Glen_b
    Mar 15, 2016 at 1:57
  • $\begingroup$ stats.stackexchange.com/questions/47498/… $\endgroup$
    – Shijia Bian
    Mar 18, 2016 at 7:01
  • $\begingroup$ The link above might be helpful $\endgroup$
    – Shijia Bian
    Mar 18, 2016 at 7:01

3 Answers 3

Reset to default
1
$\begingroup$

A simple yet robust approach could be to compute the difference between medians and then compute 95% confidence interval of such statistic with bootstrap (percentile). Such confidence interval would be adequate for inference.

You can find here some useful references:

http://r.789695.n4.nabble.com/CI-for-the-median-difference-td4399508.html

http://r.789695.n4.nabble.com/CI-for-the-median-difference-td4399508.html

http://www.statalist.org/forums/forum/general-stata-discussion/general/564770-hypothesis-testing-for-bootstrapped-differences-in-medians-in-a-randomized-clinical-trial

Yet, the small samples (5 cases in one group), limits substantially external validity, even if your inferential estimates were quite precise (anyway very unlikely).

Improve this answer
$\endgroup$
1
$\begingroup$

normal data, uneven sample sizes: Welche's t-test expects normal distribution, but allows for uneven samples sizes (and unequal variance between groups). For your samples with normal distribution, but uneven sample sizes, this test will likely give the most power. https://en.wikipedia.org/wiki/Welch%27s_t_test

non-normal data, uneven sample sizes: Welche's t-test is also suitable to use with ranked data, which converts the test to a non-parametric test. If you are heavily concerned about uneven sample sizes, this may be a good candidate for your samples with non-normal distribution. Here, you rank order transform the data before running Welche's t-test. That being said, the Mann Whitney test (ranksum) you've been using already, is non-parametric, and should not be biased by uneven sample sizes. And so you will likely find that Welche's ranked t-test closely approximates the results given by a ranksum.

However (as others have mentioned), you may have a larger issue to solve, in that your one sample size is just 5 participants. A group this small will require a substantial effect size (mean difference) in-order to be found to be significant. Before proceeding further, it may be worth conducting a power analysis, in order to determine how many participants would be required to detect your hypothesized effect.

Improve this answer
$\endgroup$
0
$\begingroup$

I suggest Approximate Randomization (AR). It applies to test statistics following any distribution and I read somewhere that it is evaluated to be either equally accurate or more accurate than other methods that assume certain distributions even when their assumptions are met. Another nice thing about AR is that it immediately spits out the $p$-value (no need to lookup distribution tables as in $t$-test).

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Approximate Randomization (I have never seen this name for it) are known as permutation tests en.wikipedia.org/wiki/… $\endgroup$
    – Jacques Wainer
    Mar 13, 2016 at 4:43
  • $\begingroup$ @JacquesWainer close (but not exactly true). Approximate randomization tests can be called "approximate permutation tests", which is different than "permutation tests". It's correct alternative names are mentioned in the same wiki page that you have linked, but in a different section: en.wikipedia.org/wiki/… -- the problem with permutation tests is that they can be very expensive, thus we can use approximate permutation tests (I call approximate randomization) which are asymptotically equal to it, but computationally more doable. $\endgroup$
    – caveman
    Mar 13, 2016 at 4:53
  • 2
    $\begingroup$ I doubt there would be any computational problems with fewer than 30 data points. $\endgroup$
    – dsaxton
    Mar 15, 2016 at 22:33
  • $\begingroup$ @dsaxton, can you say why please? I forgot the reasonable limit of data points before the Fisher exact test becomes too demanding. I think it's combinatoric, but I forgot exactly the factors that play in the speed of the combinatoric growth of the test. $\endgroup$
    – caveman
    Mar 17, 2016 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.