I have tests done looking at the brain activity in 12 different regions of the brain. It is a between subject design where there is a treatment group (N = 8) and control group (N = 14). Each participant had these scans done on the 12 different brain regions. Every effect for all 12 brain regions showed that the treatment group had higher brain activity than the control group. However, the problem is because of the low sample size only 2 of these effects were significant with the others p values somewhere between .05 and .20. From this is there any statistical analysis I can do to be able to say that overall the brain activity was higher in the treatment group compared to the control? I tried taking an average activity among the 12 brain regions for each individual and analyzing that but the sample size was still N = 8 and N = 14 so it wasnt significant. Would it be possible to combine all of the data together and do a t test on that making it N = 8*12 = 96 for the treatment group and N = 14 * 12 = 168 for the control?
2 Answers
There seem to be at least two important difficulties:
(1) Small sample sizes. Given that you could afford to look at 22 subjects, a (slightly) more efficient design would have been to have 11 in each group. But it may just be easier to enroll Control subjects than Treatment subjects.
(2) Lack of independence. It is not clear whether results from various regions of the brain are independent. If the tests at 12 regions were independent, then having all 12 go in the same direction would be convincing. Without independence, the exact suggestion in your last sentence seems off the table.
Some suggestions:
Although the 5% significant level may be crucial for journal editors, that is an arbitrary criterion. It may be worthwhile doing further investigations when P-values are around 10%.
You don't say anything about looking at correlations of results between brain regions. It may be useful to do so. If results in some brain regions might, in effect, be used to predict results in other regions, then (a) that may be important information in itself, or (b) maybe focusing on regions that provide useful independent information will permit looking at fewer regions and, over the long run, make larger sample sizes feasible.
You have two categories of subjects, Treatment and Control. You might try discriminant analysis using all 12 regions to see how effectively your data separate the two groups. Also, you might try some sort of discriminant or cluster analysis to see if the data identify known patient 'groups' or suggest unexpected ones in 12-dimensions. Maybe someone else on this site could offer an opinion whether 22 subjects is likely to be enough to make such approaches worthwhile.
Notes: Perhaps, the best known example of discriminant analysis is Fisher's introductory one. For 50 specimens of each of three varieties of iris flowers, four measurements were made: sepal and petal lengths and widths. His analysis showed that it is possible to classify almost all 150 specimens as to variety based only on these measurements.
Can you classify your subjects as to Treatment and Control using your 12 measurements? Distinct classification is more difficult than barely detecting a difference in group means. Also, are there obvious sub-categories within Treatment and/or Control groups that might be worthwhile looking at?
By contrast, cluster analysis seeks out distinct groups (perhaps not previously anticipated) based on data.
Because your tests are not independent, you need to apply Bonferroni correction (if you didn't already).
Not having "significant" effects with such a small sample sounds right to me. You just get many false positives in such cases. Don't interpret a too large p value as a "bad executed experiment". These values are a safeguard for you to indicate observations that may well be random.
Recent observations in medicine (e.g., why most published research findings are false) indicate that even a "significant" p-value must only serve as encouragement to rerun the experiment on a larger scale, unfortunately.
Don't hack your p-values to become "significant"!
So in the end, maybe the correct conclusion is that you observed higher brain activity, but you cannot rule out it was by chance because of the few patients. There is probably a 10% chance to observe such an effect by chance, and you don't just get this down to 5%.
