Small and unbalanced sample sizes for two groups - what to do? I have data for two groups (i.e. samples) I wish to compare but the total sample size is small (n = 29) and strongly unbalanced (n = 22 vs n = 7).
These data are logistically difficult and expensive to collect, so while 'collect more data' as an obvious solution isn't helpful in this case.
A number of different variables were measured (departure date, arrival date, duration of migration etc.) so there are multiple tests, some of which the variances are very different (the smaller sample having higher variance).
Initially a colleague ran t-tests on these data, and some were statistically significant with P<0.001, another was not significant with P=0.069. Some samples were normally distributed, others were not. Some tests involved large departures from 'equal' variances.
I have several questions:


*

*are t-tests appropriate here? If not, why? Does this apply only to tests where assumptions of normality and equality of variances are satisfied?

*what is a suitable alternative(s)? Perhaps a permutation test?

*unequal variance inflates the Type I error, but how? and what effect does the small, unbalanced sample size have on Type I error?

 A: T-tests that assume equal variances of the two populations aren't valid when the two populations have different variances, & it's worse for unequal sample sizes.  If the smallest sample size is the one with highest variance the test will have inflated Type I error). The Welch-Satterthwaite version of the t-test, on the other hand, does not assume equal variances.  If you're thinking of the Fisher-Pitman permutation test, it too assumes equal variances (if you want to infer unequal means from a low p-value).
There are a number of other things you might want to think about :
(1) If the variances are clearly unequal are you still so interested in a difference between the means?
(2) Might effect estimates be of more use to you than p-values?
(3) Do you want to consider the multivariate nature of your data, rather than just making a series of univariate comparisons?
A: First, as Scortchi already point out, the T-test is not suited that well to your data, because of its assumptions on the distribution of the data.
To your second point, I would propose an alternative to the T-test. If your interest is only about the fact, if the distributions of your two samples are equal or not, you could also try to use the two-sided version of the Wilcoxon rank-sum test. The Wilcoxon rank-sum test is a non-parametric test. This kind of test is especially helpful, if you are not sure about the underlying distribution of your data.
It exists an exact solution of the test for small sample sizes as well as for large cohorts. In addition, there exists also an R package which realizes the Wilcoxon rank-sum test.
Since it is a parameter free test and also handles small sample sizes, the test should suit well for you test case.
