I recently completed a study whereby I randomly assigned participants to one of two treatment groups. I tested participants at baseline, immediately post-intervention, 1 months, and 4 months on a somewhat large number of outcome variables. I was planning on running several mixed ANOVAs to examine group x time interactions. Some of the comparisons will be 2 (group) x 2 (time: baseline and post-intervention) comparisons and some will be 2 (group) x 3 (time: baseline, 1 month, 4 month) comparisons.
Before beginning my analyses, I compared the two treatment groups on all baseline variables. I found that the groups differ on 4 baseline variables if I use an alpha level of .05 or 2 baseline variables if I use an alpha level of .01 to compare the groups.
I have two questions about this:
What alpha level should I be using to compare the groups at baseline? I was thinking an alpha level of .01 because I am comparing the two groups on 24 baseline characteristics and I thought I should chose a more stringent alpha level than .05 to reduce family-wise error rate seeing as a large number of tests are being performed, but from my readings it seems most people use .05. What do you recommend?
What do I do about these differences? I could include these variables as covariates, but my sample size is quite small and using 4 covariates does not seem appropriate (which is also partly why I am favouring only accepting differences if they are significant at the .05 level)
Any help on this would be very much appreciated!