Formula for a two-sample t-test where each group is made up of multiple pooled groups

Question

I am working through Chapter 13 of the 2nd edition of An Introduction to statistical learning (the R version, see here), on Error Correction.

On page 569, in the section discussing the Scheffe method, the authors discuss testing the null hypothesis that the returns from two fund managers, one and three, in a fictional scenario are the same as the returns from three other fund managers, two, four, and five. These are the data presented in the book.

Here is the passage describing the proposed test.

.

The line that intrigues me is

"It turns out that we could test [this null hypothesis] using a variant of the two-sample t-test presented in 13.1, leading to a p-value of 0.004"

The authors do not say what this variant is or how it works. I was not aware you could pool multiple groups into a single group for each group in a two-sample t-test. The equation 13.1 that they refer to, for the two-sample t-test, is here

The numerator for the t-statistic equation is straightforward enough to adapt to the example: you simply average the mean returns from managers one and three to get one group mean, then do the same for the returns from managers two, four, and five to get the second.

What I don't know how to do is adapt the equation for $s$ in the denominator (I assume the n's for each group are the sum of the n's for each of the two and three managers. i.e. 2 x 50 = 100 for the first group and 3 x 50 = 150 for the second group).

Does anyone know the formula for this type of t-test?

Answer 1

You can kind of get the gist of how to do this by downloading the data used in this example from the package {ISLR2}. If you download that data and group managers 1 and 3 versus 2, 4, and 5 you get the following mean and standard deviations.

library(tidyverse)

fund.mini <- ISLR2::Fund[, 1:5]

mngr <- c('Manager1', 'Manager3')

d <- fund.mini %>% 
     pivot_longer(Manager1:Manager5, names_to = 'manager', 
     values_to = 'returns') %>% 
     mutate(grp = if_else(manager %in% mngr, 1, 0))

md <- d %>% 
  group_by(grp) %>% 
  summarise(
    xbar=mean(returns),
    sds=sd(returns)
  )
md

# A tibble: 2 × 3
    grp  xbar   sds
  <dbl> <dbl> <dbl>
1     0 0.233  6.74
2     1 2.9    7.45

From here, you could do the t-test as written and obtain a p value close to what is quoted.

Obviously, we only have the summarized data, so the question is how do we get the means and standard deviations just from the summarized data?

The means are easy. Because each manager has 50 observations, you can just take the mean of the means. If they all had a different number of observations, then you'd need to do a weighted average.

xbar <- c(3, -0.1, 2.8, 0.5, 0.3)
s <- c(7.4, 6.9, 7.5, 6.7, 6.8)
ix <- c(1, 3)

mean(xbar[ix])
[1] 2.9
mean(xbar[-ix])
[1] 0.2333333

What about the standard deviations? Well ,you can use the pooling trick shown in your equation for the t test to get those.

# Two means are estimated, so 98 in denominator
sqrt(sum((50-1)*s[ix]^2)/98)
[1] 7.450168

# three means are estimated, so 147 in denominator
sqrt(sum((50-1)*s[-ix]^2)/147)
[1] 6.80049

They are close, not exact for some reason I can't think of.

Now you have the statistics you need to do the t test. Simply use the formula as written. Because I'm a bit lazy, here is a python call to a function which computes the t-test from sample statistics

from scipy.stats import ttest_ind_from_stats

ttest_ind_from_stats(0.233, 6.8, 150, 2.9, 7.45, 100)
>>>Ttest_indResult(statistic=-2.9233789202479876, pvalue=0.0037826444901180602)

P value is right where it should be, and is close to what you would get if you computed the test from the raw data

> summary(lm(returns ~ grp, data=d))

Call:
lm(formula = returns ~ grp, data = d)

Residuals:
     Min       1Q   Median       3Q      Max 
-16.4490  -4.9636  -0.3567   4.9589  16.9262 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept)   0.2333     0.5741   0.406  0.68477   
grp           2.6667     0.9077   2.938  0.00362 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 7.031 on 248 degrees of freedom
Multiple R-squared:  0.03363,   Adjusted R-squared:  0.02973 
F-statistic: 8.631 on 1 and 248 DF,  p-value: 0.003617

Answer 2

Thanks to @Demetri Pananos for getting me started on how to calculate this.

The equation in the ISLR2 book is

So, following this, if we call the combined returns of managers 1 and 3 group 1, and the combined returns of managers 2, 4, and 5 group 2.

xbar <- c(3, -0.1, 2.8, 0.5, 0.3) # mean returns managers 1-5
s <- c(7.4, 6.9, 7.5, 6.7, 6.8) # sd returns managers 1-5
ix <- c(1, 3) # index for group 1

The part of the equation I was having trouble adapting from the simple two-group t-test to the groups-within-each-group t-test was the numerator of the fraction in the equation for $s$ prior to applying the square root, specifically the $(n_t - 1)s_t^2 + (n_c - 1)s_c^2$ part: because we have multiple $i$ groups in both $(n_t - 1)s_t^2$ and $(n_c - 1)s_c^2$ rather than a single group. This is where @Demetri Pananos' answer helped so much: for each of these two group variances we simply sum the variances of the fund managers within each group

# group 1 has two managers
(50-1)*s[ix]^2  # two individual variances, one for each manager, before summing
# output
[1] 2683.24 2756.25

sum((50-1)*s[ix]^2) # pooled variance for first group after summing
# output
# [1] 5439.49


# group 2 has three managers
(50-1)*s[-ix]^2 # three individual variances before summing
# output 
[1] 2332.89 2199.61 2265.76

sum((50-1)*s[-ix]^2) # pooled variance for second group after summing
# output
# [1] 6798.26

So, now we know how to get the two pooled variances for each of the groups in the t-test, we can calculate the numerator of $s$ equation before we apply the square root

numerator_seq <- 5439.49 + 6798.26

For the denominator of the $s$ equation before square root we take total number in group 1 plus total number in group 2 minus 5 (this is the part I guessed: number of managers in group 1 plus number of managers in group 2: 2 + 3 = 5)

denom_seq <- 100 + 147 - 5

Now we can calculate $s$

s <- sqrt(numerator_seq/denom_seq)

And now that we have $s$ we can calculcate the denominator of the t equation

denom_t <- s * sqrt(1/100 + 1/150)

Numerator of the t equation is the average of mean returns for all of the fund managers in each group

num_t <- mean(c(3,2.8)) - mean(c(-0.1,0.5,0.3))

Finally we can calculate t

t <- num_t/denom_t

t

# output
# [1] 2.9047

And we can get the p-value from plugging our t statistic values into a t distribution with n_group1 + n_group2 - 5 (5 from number of managers in group1 = 2 plus number of managers in group2 = 3) = 145 degrees of freedom

p <- 2*pt(q = t, 
          df = 100 + 150 - 5, 
          lower.tail = FALSE)

p

# output
# [1] 0.004011974

This yields the same p-value as is supplied in ISLR.