11
$\begingroup$

I have data for two groups (i.e. samples) I wish to compare but the total sample size is small (n = 29) and strongly unbalanced (n = 22 vs n = 7).

These data are logistically difficult and expensive to collect, so while 'collect more data' as an obvious solution isn't helpful in this case.

A number of different variables were measured (departure date, arrival date, duration of migration etc.) so there are multiple tests, some of which the variances are very different (the smaller sample having higher variance).

Initially a colleague ran t-tests on these data, and some were statistically significant with P<0.001, another was not significant with P=0.069. Some samples were normally distributed, others were not. Some tests involved large departures from 'equal' variances.

I have several questions:

  1. are t-tests appropriate here? If not, why? Does this apply only to tests where assumptions of normality and equality of variances are satisfied?
  2. what is a suitable alternative(s)? Perhaps a permutation test?
  3. unequal variance inflates the Type I error, but how? and what effect does the small, unbalanced sample size have on Type I error?
Improve this question
$\endgroup$

2 Answers 2

Reset to default
11
$\begingroup$

T-tests that assume equal variances of the two populations aren't valid when the two populations have different variances, & it's worse for unequal sample sizes. If the smallest sample size is the one with highest variance the test will have inflated Type I error). The Welch-Satterthwaite version of the t-test, on the other hand, does not assume equal variances. If you're thinking of the Fisher-Pitman permutation test, it too assumes equal variances (if you want to infer unequal means from a low p-value).

There are a number of other things you might want to think about :

(1) If the variances are clearly unequal are you still so interested in a difference between the means?

(2) Might effect estimates be of more use to you than p-values?

(3) Do you want to consider the multivariate nature of your data, rather than just making a series of univariate comparisons?

Improve this answer
$\endgroup$
7
  • $\begingroup$ Hi Scortchi, thanks for your reply. I've considered the questions you posed: $\endgroup$
    – DeanP
    Dec 12, 2012 at 7:06
  • 2
    $\begingroup$ (1) Both the variance and mean can be informative for our study (e.g. migration departure dates may be significantly later for one population AND the range in departure dates is more variable). $\endgroup$
    – DeanP
    Dec 12, 2012 at 7:12
  • 3
    $\begingroup$ (1) Just mentioned it because people often view unequal variances solely as a technical problem & forget it's an interesting fact in its own right. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 12, 2012 at 9:19
  • 2
    $\begingroup$ (2) My point was more that a list of p-values is generally less useful than a list of effect size estimates (which could be means, medians, variances, or whatever) with confidence intervals. Especially with small samples, confidence intervals can show whether effect sizes of practical importance are still concordant with the data even when the p-value is high. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 12, 2012 at 9:22
  • 2
    $\begingroup$ (3) I was thinking of one independent variable (group) & several dependent variables (migration time &c.): an interesting difference between groups might be a change in the relationship between dependent variables. A first step would be a nice matrix with boxplots or dotplots comparing each d.v. between groups along the diagonal, & scatterplots for each pair of d.v.s (again distinguishing groups) in the other cells. And to be honest, for an exploratory analysis with small sample sizes, that could well be the last step. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 12, 2012 at 9:24
1
$\begingroup$

First, as Scortchi already point out, the T-test is not suited that well to your data, because of its assumptions on the distribution of the data.

To your second point, I would propose an alternative to the T-test. If your interest is only about the fact, if the distributions of your two samples are equal or not, you could also try to use the two-sided version of the Wilcoxon rank-sum test. The Wilcoxon rank-sum test is a non-parametric test. This kind of test is especially helpful, if you are not sure about the underlying distribution of your data.

It exists an exact solution of the test for small sample sizes as well as for large cohorts. In addition, there exists also an R package which realizes the Wilcoxon rank-sum test.

Since it is a parameter free test and also handles small sample sizes, the test should suit well for you test case.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.