Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have data for two groups (i.e. samples) I wish to compare but the total sample size is small (n = 29) and strongly unbalanced (n = 22 vs n = 7).

These data are logistically difficult and expensive to collect, so while 'collect more data' as an obvious solution isn't helpful in this case.

A number of different variables were measured (departure date, arrival date, duration of migration etc.) so there are multiple tests, some of which the variances are very different (the smaller sample having higher variance).

Initially a colleague ran t-tests on these data, and some were statistically significant with P<0.001, another was not significant with P=0.069. Some samples were normally distributed, others were not. Some tests involved large departures from 'equal' variances.

I have several questions:

  1. are t-tests appropriate here? If not, why? Does this apply only to tests where assumptions of normality and equality of variances are satisfied?
  2. what is a suitable alternative(s)? Perhaps a permutation test?
  3. unequal variance inflates the Type I error, but how? and what effect does the small, unbalanced sample size have on Type I error?
share|improve this question
add comment

1 Answer

T-tests that assume equal variances of the two populations aren't valid when the two populations have different variances, & it's worse for unequal sample sizes. If the smallest sample size is the one with highest variance the test will have inflated Type I error). The Welch-Satterthwaite version of the t-test, on the other hand, does not assume equal variances. If you're thinking of the Fisher-Pitman permutation test, it too assumes equal variances (if you want to infer unequal means from a low p-value).

There are a number of other things you might want to think about :

(1) If the variances are clearly unequal are you still so interested in a difference between the means?

(2) Might effect estimates be of more use to you than p-values?

(3) Do you want to consider the multivariate nature of your data, rather than just making a series of univariate comparisons?

share|improve this answer
    
Hi Scortchi, thanks for your reply. I've considered the questions you posed: –  Dean Portelli Dec 12 '12 at 7:06
1  
(1) Both the variance and mean can be informative for our study (e.g. migration departure dates may be significantly later for one population AND the range in departure dates is more variable). –  Dean Portelli Dec 12 '12 at 7:12
1  
(1) Just mentioned it because people often view unequal variances solely as a technical problem & forget it's an interesting fact in its own right. –  Scortchi Dec 12 '12 at 9:19
1  
(2) My point was more that a list of p-values is generally less useful than a list of effect size estimates (which could be means, medians, variances, or whatever) with confidence intervals. Especially with small samples, confidence intervals can show whether effect sizes of practical importance are still concordant with the data even when the p-value is high. –  Scortchi Dec 12 '12 at 9:22
1  
(3) I was thinking of one independent variable (group) & several dependent variables (migration time &c.): an interesting difference between groups might be a change in the relationship between dependent variables. A first step would be a nice matrix with boxplots or dotplots comparing each d.v. between groups along the diagonal, & scatterplots for each pair of d.v.s (again distinguishing groups) in the other cells. And to be honest, for an exploratory analysis with small sample sizes, that could well be the last step. –  Scortchi Dec 12 '12 at 9:24
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.