What statistical tests would you recommend for group difference for my data?

Question

I am trying to find out what statistical tests to run on a set of aggregate data (no individual data) on 3 different groups, variables are summarized below: Group 1 in site 1: Mean value (blood sugar) of the total sample before treatment, and mean value (blood sugar) of the same group after treatment Group 2 in site 2: same as above Group 3 in site 3: same as above

I want to know if it makes sense to run the following analysis on the above data:

1) Paired t-test on each group separately for the group difference in blood sugar means after treatment and before treatment (to test if any increase or decrease in the whole group average is statistically significant) 2) where I start struggling is when I think of the tests to apply for the difference between the groups. I can calculate absolute difference in the means after and before treatment for each group, have then difference of the means after and before in group1, difference of the means after and before in group 2 and same for group 3. Can I run an ANOVA on the difference means between the three groups (mean diff of means after and before group 1 vs mean difference of the means after and before for group 2 vs the one of group 3)?? If yes what other data do I need?

Answer 1

I have edited my answer to correct a mistake.

Use ONE paired t-test to see whether the treatment had any effect on mean blood sugar across the three sites. You cannot test whether there are differences among the sites.

Here is some example data so you can check your calculations against mine (if you need to).

--------------Before   After
Site 1         0.23     0.31
Site 2         0.05     0.11
Site 3         0.17     0.24

The outputs from the t-test are:

• t-statistic = 12.124
• degrees of freedom = 2
• p-value = 0.006734

So, for this data we would say there is a statistically significant effect of treatment.

Aggregated data has much less information than the raw data. This means you can answer fewer questions with it. For example, you will not be able to test whether the treatment has different effects at the different sites (this is known as an interaction effect). The aggregated data has eliminated the replication (the 1000 patients/subjects) within sites that would have allowed you to do this.

For this reason you also cannot test whether there are any differences among the sites. Although you have two values for each site, they are not independent of each other, and it's equivalent to saying you only have one value per site. In general, you need replication (more than one independent value per site) to test for differences between groups.

In your research, you may come across two-way ANOVA without replication. It is used for the situation where there is only one value in each combination of factor levels (Site1 Before, Site1 After, Site2 Before ...) just like in the example data above. It will test whether there is an effect of treatment and also test whether the sites are different. But you must not use this! It assumes that the values in the before and after columns are independent of one another (from different groups of people, not the same group of people). The p-values will be wrong, and therefore any conclusions you make will also be wrong!

In my original answer I suggested that using pearson's correlation coefficient would be suitable to test for differences between sites because it deals with paired data points. But this also suffers from elevated Type I errors (unreliable p-values). There is no way to know whether any differences between sites are due to random sampling error or whether they represent additional site to site variation.