What is the best way to combine data from two sessions for a variable for two independent groups before performing statistical tests between groups?

Question

We have data on a physiological variable of interest (Metabolic Cost of Walking) from 2 groups of subjects (10 young adults and 10 old adults). We measured each one of them twice, once in the morning and once in the afternoon of the same day. We did this to check for variations in metabolic data over the course of the day, both within and between the 2 groups.

Now we want to combine the data from morning and afternoon per group and then test the differences in the mean values on a group level between the young and the old group through a statistical test. We intend to perform a between-subjects, independent test.

What is the best way to combine the data (either at baseline or statistically) from morning and afternoon per group and then perform such a test ?

Answer 1

It is impossible to say how to combine AM and PM scores--or what to do after that--without knowing your motivation for the study and having a more-focused reason for doing both AM and PM tests. 'Checking for variations' is about as vague as it gets. Why are you really going to the extra effort to do both tests? How do you suppose they may differ?

If the overall purpose is to establish the semi-obvious fact that younger people have more 'energy' than older ones, then 10 subjects in each age group is probably too few to give results of interest, no matter what P-values you get. If you have a particular sub-population of interest (e.g., people in a city with a particular kind of water pollution), then results may be less predictable and more interesting, even with small samples.

I can see arguments for using any one of four methods of 'combining' AM and PM scores: (a) average, (b) difference, (c) worst, or (d) best score.

If you are familiar with 'metabolic cost' scores, then you might know how consistent you expect AM and PM scores to be. You might also have an idea whether such scores tend to be roughly normally distributed across a large population.

If you are at the exploratory phase of using these scores, you might try four separate 2-sample t tests after combining in each of the four ways (a)-(d). With sample sizes as small as 10 in each group. I hope data are nearly normal so that t tests are appropriate for comparing Young vs. Older, because the power of nonparametric tests, such as 2-sample Wilcoxon test, is somewhat lower, especially using such small groups.

You should guard against false discovery doing all four tests, perhaps looking for P-values below 1% or 2% to reject.

If you are familiar with these scores and feel it is OK to assume near normality, then you might consider an appropriate partially hierarchical ANOVA design, which would include all effects. Then if warranted, you could do ad hoc tests. Some of these ad hoc tests might show which of (a)-(d) is getting at the truth. Initial rejection of the overall ANOVA model as a prerequisite for doing ad hoc tests offers some protection against false discovery.

Notes: (1) If I understand your experiment correctly, a possible ANOVA model for a design that includes all effects, and does not combine AM and PM scores for individuals, is as follows:

$$Y_{ijk} = \mu + \alpha_i + \tau_j + \{\alpha\tau\}_{ij} + S(\alpha)_{k(i)} + e_{ijk},$$ where $i = 1,2$ age groups, $j=1,2$ times of the day, $k(i) = 1, 2, \dots, 10$ randomly chosen subjects within each age group, $S(\alpha)_{i(i)} \stackrel{iid}{\sim} \mathsf{Norm}(0, \sigma_S),$ and $ e_{ijk} \stackrel{iid}{\sim}\mathsf{Norm}(0, \sigma).$

You could study age, diurnal effect, and their interaction with lines $\alpha, \tau, \{\alpha\tau\}$ of the ANOVA table. These are all fixed effects. 'Subject' is a random effect. A three-way interaction is not supported because of the nesting. Parentheses $(\;)$ indicate nesting and are read as 'within'.

(2) Here is one reason parts of my discussion focus on having only ten subjects in each group.

Suppose, for normal data, that you are trying to detect a difference of one standard deviation with 10 subjects in each of two groups. For example, this might be the difference between $\mathsf{Norm}(\mu=100,\sigma=15)$ and $\mathsf{Norm}(\mu=115,\sigma=15).$ Then a computation using a noncentral t distribution shows the power is only about 56%.

With the same kind of data, using 2-sample Wilcoxon test, a simulation (with R) shows that the power is only about $0.511 \pm 0.003.$ Even if the effect is present, you have only about a 50:50 chance of detecting it.

set.seed(2020)
pv = replicate( 10^5, 
      wilcox.test( rnorm(10,100,15), rnorm(10,115,15) )$p.val )
mean(pv < .05)
[1] 0.51138        # aprx power for 5% level test
2*sd(pv < .05)/sqrt(10^5)
[1] 0.0031430  

mean(pv < .02)
[1] 0.36578        # aprx power for 2% level test

Addendum in response to question in comment:

• Suppose the main difference between Young and Older people is that Older ones have higher metabolic cost later in the day. But Younger people stay steady throughout the day. Would that be of interest? If so, then look at PM/PM difference. (Either order, AM - PM or PM - AM, but be consistent.)

• What if most efficient score is the 'real' one and some people are occasionally less efficient? (Brief headache, upset over bad news, today's pizza lunch not digesting properly.) Then use most efficient score.

• What if least efficient sore is more reliable? (Anyone can happen to have an occasional atypical efficient score, but that's an anomaly.) Then use least eff. score.

I don't suppose you're limited to using just one of the ways of summarizing data.

I really have no idea which to use because this is not my area of study. Presumably someone familiar with these scores would have a clue what they really mean. And presumably someone not yet familiar with them would want to find out before using them in a study.

I hope you also pay attention to the issue of potentially low power to detect real effects due to the small sample sizes you mentioned.

Answer 2

So we are interested to see if there is a diurnal effect for either of the groups or for both the groups. In addition, we are also interested to see whether there are any age-group effects and an interaction effect between the time of the day (AM and PM) and the age-groups (Young and Old)

So why combine the AM and PM data at all? First, a single model will give you all those results (assuming that you have adequate power, as @BruceET rightly is concerned about*). Second, if there is an interaction between time of day and age group then there really is no good way to combine AM with PM data.

A simple model could be:

MCW ~ ageGroup*timeOfDay

where MCW is the measured metabolic cost of walking, and timeOfDay is AM or PM. Say that "Young" and "AM" are the reference values for the categorical predictors. Then you get 4 coefficients reported:

1. An intercept, representing the estimated MCW for Young in the AM.

2. A coefficient for ageGroup, representing the difference between Old and Young in the AM;

3. A coefficient for timeOfDay, representing the difference between PM and AM for the Young group;

4. An interaction-term coefficient, representing the difference in the timeOfDay coefficient between the Old and Young groups (and also the difference in ageGroup coefficient between PM and AM).

Then there are 2 general possibilities.

First, if the interaction term is too small to be of interest, then you can just go to an additive model:

MCW ~ ageGroup + timeOfDay

and the ageGroup coefficient gives you the result you want for Old-Young difference.

Alternatively, if the interaction term is large enough to be of interest then there is no good way to combine the AM and PM values. You will have shown that the PM-AM difference depends on the ageGroup. You will have one ageGroup difference for AM and a different one for PM. So any attempt to combine AM and PM values across age groups will be misleading at best. The estimated MCM values for the 4 combinations of ageGroup and timeOfDay will better illustrate your results.

---

*Using a mixed model might help with power. For example, using the R lmer() syntax you could model:

MCW ~ ageGroup*timeOfDay + (1|subject),

which allows for differences in intercept terms among your 20 participants in a way that might help improve the precision of estimates for the regression coefficients.