# When to perform a hypothesis test on difference of means?

I'm reading a research paper found at this link.

Although overall spending was higher in traditional plans than HDHPs for both populations, OOP spending was substantially higher for those in HDHPs compared with traditional plans, especially for those with SU/MH disorders (\$573 vs \$280).

Is this a two-sample t-test that was performed? Also, why is it that they performed the test (I'm assuming) on the difference of the means and not something like the sum?

• We can't see the research paper - the link points to a local file on your computer. Commented Jan 2, 2023 at 0:24
• @DaveArmstrong Probably jamanetwork.com/journals/jamapsychiatry/fullarticle/2763800 (click on Figures/Table to see this table) Commented Jan 2, 2023 at 0:35
• @DaveArmstrong Yes that's the one. Updated the post. Commented Jan 2, 2023 at 0:59
• Your math formatting is off but it's too few characters to edit. Commented Jan 2, 2023 at 19:25

#### Is this a two-sample t-test that was performed?

The authors of this article did not do a great job of explaining explicitly what they did. Clearly they looked at differences of means, they provided a significance cutoff, and they made claims about which were significant by denoting them with superscripts in the table. They don't, however, provide anything resembling t-statistics, specific p values, effect sizes, confidence intervals, or even what the sample standard deviations are (though it seems implied by the parenthetical values beside the means in the article's table). So while we can only guess that's what they did, they were not very straightforward in elucidating that point.

In order to obtain a t-statistic without the raw data, we can apply the following formula if we know the values that belong in each:

$$t = \frac{\bar{x}_1-\bar{x}_2}{\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}}$$

where $$\bar{x}$$ equals the mean of a given sample, $$s$$ equals the standard deviation of a given sample (and by extension $$s^2$$ is its variance), and $$n$$ equals the sample size of a given sample. If we look at Row 1 as an example:

we can assume that:

• $$\bar{x}_1 = 7,924$$
• $$\bar{x}_2 = 7,012$$
• $$s_1 = 15,971$$
• $$s_2 = 15,285$$
• $$n_1 = 27,485,702$$
• $$n_2 = 1,624,961$$

From there, we would just plug in the values like so:

$$t = \frac{7,924-7,012}{\sqrt{\frac{15,971^2}{27,485,702}+\frac{15,285^2}{1,624,961}}}$$

You can calculate this by hand, or you can do so manually in R like so:

#### Manually Calculate T ####
m1 <- 7924
m2 <- 7012
s1 <- 15971
s2 <- 15285
n1 <- 27485702
n2 <- 1624961
se <- sqrt((s1^2/n1) + (s2^2/n2))
t <- (m1 - m2)/se
t


Either way, you should get a substantial t value of $$73.71717$$ (this isn't entirely surprising given the sheer sample size given in the article). Most t-tables do not have degrees of freedom that approach the sample sizes we have here, but we can very safely assume that a t value and sample this high would give you a significant p value as well. If you want to approximate the p value in R directly, one can also use the pt function, which approximates a Student t distribution with which one can obtain a p value from.

p <- 2*pt(t,
n1 + n2 -2,
lower.tail=F)


Which gives us zero (or at least an extremely low value). I have created a function in R that can combine all of these steps into a more readable summary below, wherein you can either manually input the values yourself or use the values we saved previously:

twotail_t_test <- function(m1, m2, n1, n2, s1, s2){
mean.diff <- abs(m1-m2)
se <- sqrt((s1^2/n1) + (s2^2/n2))
t <- mean.diff/se
p <- 2*pt(t, # use t value
n1 + n2 -2, # subtract for degrees of freedom
lower.tail=F) # two tailed test
cat("-------------------------------------",
"\nIndependent Sample T-Test Calculator",
"\n-------------------------------------",
"\nSample Size 1:",n1,
"\nSample Size 2:",n2,
"\nMean 1:",m1,
"\nMean 2:",m2,
"\nSD 1:",s1,
"\nSD 2:",s2,
"\n----------------------------------",
"\nMean Difference:",round(mean.diff,3),
"\nStandard Error:",round(se,3),
"\nT Value:",round(t,3),
"\nP Value:",round(p,3))
}

twotail_t_test(m1,m2,n1,n2,s1,s2)


Which should give you a summary of everything we have discussed so far:

-------------------------------------
Independent Sample T-Test Calculator
-------------------------------------
Sample Size 1: 27485702
Sample Size 2: 1624961
Mean 1: 7924
Mean 2: 7012
SD 1: 15971
SD 2: 15285
----------------------------------
Mean Difference: 912
Standard Error: 12.372
T Value: 73.717
P Value: 0


You can fiddle with it any way you please, such as the code twotail_t_test(10,19,50,50,51,21), which gives you these values at the end of the summary:

Mean Difference: 9
Standard Error: 7.8
T Value: 1.154
P Value: 0.251


#### Side Note on This Study

While this can all be estimated, there are still a number of issues present. The most obvious is the substantial differences in sample sizes, as the traditional group is almost 27x the size of the HDHP group. There is also no effect size approximating the effect. Because the variance seem different from each other, it may be good to estimate Hedge's g, which can be derived from this formula:

$$g = \frac{ \bar{x}_1-\bar{x}_2 } { \sqrt{\frac{(n_1-1) \times s_1^2 + (n_2-1) \times s_2^2}{n_1+n_2-2}} }$$

#### Obtain Numerator and Denominator for Hedges G ####
hedge.numer <- m1-m2

hedge.denom <- sqrt(
(((n1-1)*s1^2) + ((n2-1)*s2^2))/(n1+n2-2)
)

#### Calculate Hedges G ####
hedges.g <- round(hedge.numer/hedge.denom,3)
hedges.g


Here we find that the effect size is fairly negligible...around $$0.057$$. So while the effect seems significant, the magnitude of the effect doesn't seem considerably strong.

#### Also, why is it that they performed the test (I'm assuming) on the difference of the means and not the sum, let's say?

It's hard to guess what their specific rationale was. In the case of a t-test, t-statistics are partially defined by the mean. This is because t-distributions are supposed to resemble normal distributions, and means provide a convenient approximation of the centrality of the estimate. How different these centrally located means are from each other (in this case two sample means and their distance from each other), gives us some approximation of what the effect may be. Sum statistics don't provide any information about where the data is dispersed, only the total raw units that were derived from the sample.

A good visualization tool for why means and standard deviation matter for comparison between samples can be found at this interactive website which also shows how Cohen's d works. The dashed lines in the middle of the bell curves represent the means, and you can see how shifting these means from each other with the top slider affects the magnitude of the effects.

• Would you say that Hedge's g is analogous to Yule's Q from the chi-squared test? Commented Jan 2, 2023 at 14:22
• Practically speaking, Yule's Q and Hedge's g are both effect sizes that attempt to measure the magnitude of the studied effect, but the properties and applications of these are different. As a direct example, Yule's Q is bounded between -1 and +1, whereas Cohen's d and Hedge's g can both exceed 1. Commented Jan 2, 2023 at 14:52

Test differences in means when the interest is in how much higher or lower one mean is from the other. For instance, we might want to know if a new medical treatment results in a longer life expectancy than an older treatment. A simple analysis might test the mean life expectancy of patients who received each treatment.

• Right, but does this mean that it's a regular t-test? The paper mentions a two-sided significance test but I'm unclear on what they used. I want to recreate the test in R. t.test(573,280, alternative = "two.sided") creates an error. Commented Jan 2, 2023 at 3:34
• @Antonio As it should create an error: a standard t-test does not make sense for two values. You would input the entire data sets into those first two function arguments, not just the observed means. // If you don’t have access to the entire data set, just summary statistics, you might have to code your own implementation of the t-stat and p-value. I do not know of a function that operates on summary statistics instead of data (though I do expect one to exist).
– Dave
Commented Jan 2, 2023 at 4:18
• Correct me if I am wrong, but I thought all you needed to conduct a two-sample t-test was means for each group, variance for each group, and the sample size for each group. Wouldn't this essentially get you a t-statistic and an associated p value without the raw data? As an example: stats.stackexchange.com/a/65812/345611 Commented Jan 2, 2023 at 4:56
• From the article, it doesn't appear clear what the SD or variance is for each sample, but I assume the parenthetical numbers next to the means in the table are these values, so you would be able to produce some kind of t-statistic if these were the numbers they are supposed to be. Commented Jan 2, 2023 at 4:58
• I have provided an answer to your question, but your assumption seems correct, as it is also what I calculated. Commented Jan 2, 2023 at 8:13