# What statistics test to use to compare multiple groups with different sample sizes

I'm running a test where I need to compare four groups on different dependent variables. Two of them are categorical and I'll a use Chi-squared test for the head-count while one y is a continuous variable: Reinvestment Value. the four groups and statistics with respect to the continuous y are:

• Control group: n=2749 I don't have the statistics yet
• Treatment A n=624, mean=3413.9 s2=30269139.5 sd=5501.7, k=12, sk=3.1
• Treatment B n=38, mean=1546.62 s2=1710133.95 sd=1307.72, k=-0.81, sk=0.82
• Treatment C n=1708, mean=2528 s2=21949273.2 sd=4685, k=28, sk=4.5

just to clear up all the confusion, I have the actual samples data not just the statistics.

So I wonder what test should I use.

I wanted to use ANOVA, to compare means of my four different groups, but:

• data violate the assumption on normality, all the groups look more similar to an exponential distribution. However, one of the four, the one with smallest kurtosis, is an exponential with couple of picks
• and also they violate the assumption on equal variances proved by both Barlette's and Levene's test

I have read that as long as the sample sizes are equal and big enough I can still use ANOVA, however

• also my sample sizes are different and I have some of the groups with n>100 and others n<100. However, none of those has less than 30 observation, so I am not sure if I can still assume the central limit theorem.
• also I have two groups with kurtosis > 5 and one of them ~0. Again I read that if kurtosis < 5 we can still assume normality

So I thought to move to a non-parametric test, like Kruskal-Wallis, but also in that case I have seen that they result inefficient if the homogeneity of variances is violated.

So I guess I have multiple questions here:

1. How many n should I have to assume the central limit theorem? I read controversial answers on this, some say >30, others >50 and others>100. could I still assume the data normal with 50<n<100?
2. It looks like the homogeneous variances assumption is even more important than the assumption about normality, so what to do if data do violate this assumption? Shall I transform them? Is there a better non-parametric one to use, like the Welch test?
3. If I should transform them, how, what to use?
4. Can I randomly sample the biggest groups to get an equal n from all the groups and apply ANOVA?
5. If I have had homogeneous variances, all the other conditions stay the same, should I use a parametric ANOVA or non-parametric Kruskal–Wallis?
6. I will also need a post test to assess where the differences are, in case of ANOVA I was thinking to use Tukey, but what to use in case of Kruskal–Wallis?

Just to give context, I'm in the situation where at least 2 means, variances and standard deviation look very different so reasonably they are. Also for information, I use RStudio to run the analysis.

Hope someone can help with this,

• Would it be possible to edit your post to include your actual data? Feb 2, 2021 at 10:59
• @StephanKolassa thanks for your comment, I have added the sample statistics and the distributions, hope this can help! For the control I don't have the data yet but I know the sample size is much bigger than the other groups
– VR88
Feb 2, 2021 at 13:36

## 3 Answers

First off, ANOVA is often surprisingly robust to strange distributions. What is important is not that model residuals are normally distributed, but that parameters are (approximately) normally distributed. Here, that would be the group means. You can take a look at this by bootstrapping each group mean. I suspect that the bootstrapped means will be nicely bell-shaped, giving you a little more confidence. (The only group I am slightly concerned about is group B with only $$n=38$$.)

That said, you could run a permutation alternative to ANOVA. In R, there is RVAideMemoire::perm.anova. A useful textbook is Good's Permutation, Parametric, and Bootstrap Tests of Hypotheses. If you do so, be sure to also run the plain vanilla ANOVA alongside and compare the resulting $$p$$ values - you may well find that they are very close together. (And note that $$p$$-values that are astronomically small do not have much more content than "$$p<.0001$$".)

• Hello Stephan,I have tried what you have suggested and first thing first whe I have bootstrapped my group means and all the sampling means distributions are normally distributed, also the one with 38 observation under alpha 0.05, p-value=0.06. I have also run the permutation ANOVA ( something very new for me) along with the normal ANOVA, I guess it was Eucledean distance since F is exactly the same, however p-values differ with parametric ANOVA <.0001, while permutation ANOVA with 999 permutations=0.001 and with 9999 permutations=.0001, with 99999<.0001, thus which one to use?
– VR88
Feb 3, 2021 at 13:23
• Good to hear the results agree. (Anything else would have surprised me.) To be honest, there is little to choose between the two $p$ values. Remember that $p$ values themselves are also random variables, so they have a variance! So don't go hunting for spurious precision. Also, will such a difference in $p$ values actually make a difference in your conclusions? I wouldn't think so. Finally, I recommend Gelman & Stern (2006, TAS). Feb 3, 2021 at 13:55
• Many thanks, all this is very handy and opening a new line of action; you are right in this case not too much difference anyway still I'll have to reject the H0 which was what simple Anova was saying from the beginning. As Post hoc test would you use TukeyHSD applied on permutation Anova or I can consider Tukey on simple ANOVA reliable enough to see pairwise differences?
– VR88
Feb 3, 2021 at 14:28
• Given that the permutation and the "vanilla" ANOVA seem to agree well enough, I would be comfortable with doing post hoc tests on the vanilla ANOVA. Feb 3, 2021 at 15:30
• brilliant, many thanks for all your help!
– VR88
Feb 3, 2021 at 15:43

I will try to address 3 and also a few others affected by choice of transformation. You can try a log (base of your choice, though natural log or $$log_{2}$$ might be best) transform, but if any $$Y = 0$$ this is not optimal as you'd have to add a small amount to each value. Square-root transform will work better in this case. If you chose a transformation, try to choose a sensible one. You could also try a Box-Cox transform.

The purpose of a transformation is obviously to get your data to meet assumptions, so you can check on 2 after you choose one.

Note, that the sample-size difference is only an issue if you use formulas for ANOVA that rely on the assumption of equal n's. However, really unbalanced designs can be difficult in any setting, and it seems your design is very unbalanced (n = 38 v. n's > 1000). Noting that difficulty, in R, you could fit a regression model with your grouping variable as the factor, using the control group as the comparison group. The emmeans package will enable post-hoc testing using the model estimates. However, with the unbalanced design, the estimates will be noisy.

• Hello Rick, thanks for your comment. In my case no Y=0 per assumption, reinvestment value has to be always >0, so would the transformation leads to meet the assumption of normality or homegeneity of variances? In the case I will get homogeneity of variances shall I still move on with a non-parametric then due to the sample size unbalanced design? do you think could I fix this randomly selecting participants from the biggest groups and balancing them with the smalles one and so call it a day with an ANOVA?
– VR88
Feb 2, 2021 at 14:50
• Also I just wanted to add that after performing the Box-Cox transformation I have not met the normality assumption and variances are still very different despite the p-value has improved passing from 0.000000000000007 to 0.0000000016. I have also tried to transform through sqrt but nothing shapiro test and Kolmogorov-Smirnov still reject the null hypothesis
– VR88
Feb 2, 2021 at 17:11

Another idea is to try (multinomial) logistic regression, as in T-tests, manova or logistic regression - how to compare two groups? which is a similar problem but with two groups to compare.

This would make fewer assumptions than anova or manova.