0
$\begingroup$

I would like to compare two means

I have the following variables

group variable (IND)

  • 12 (161 observations)
  • 17 (151 observations)

output variable (pv1)

  • scale from 1 to 5

I am working with R one test is a Welch two sample t-test and the other test is a two Sample t-test The output is slightly different. I've never heard of a Welch two sample t-test. Which test should I use?

data=read.csv("teilzeitarbeit.csv",header=T,sep=";")

t.test(data$pv1~data$IND)

t.test(data$pv1~data$IND, var.equal=TRUE, data=data)

Sorry, I don't know why you can't see the tilde and dollar sign.

$\endgroup$
3
  • 1
    $\begingroup$ What are 12 and 17? $\endgroup$
    – Dave
    Aug 2, 2020 at 0:02
  • $\begingroup$ Should your code look like this: t.test(data$pv1 ~ data$IND)? $\endgroup$ Aug 2, 2020 at 0:48
  • 1
    $\begingroup$ In order to get lines of code, put four blank spaces at the start of each line. If using \$ for money and not to begin 'math mode' type \$. In math expressions (between \$-signs), use \sim to get $\sim$. $\endgroup$
    – BruceET
    Aug 2, 2020 at 1:08

2 Answers 2

1
$\begingroup$

The pooled two-sample t-test must be used only when you 'know' that the two populations have equal variances. You can 'know' because of prior experience with similar data or because you're really sure of a model for the data. [For example, that might occur when variance is almost completely due to instrument error and the same instrumentation has been used for both groups.]

However that 'knowing' should not be because you did a test for equal variances. There are several reasons for this prohibition, probably the most important of which is that variance tests have poor power and can give misleading results. [Based on simulation studies, the use of a test for equal variances as a 'screening test' to decide between pooled and Welch t tests has been deprecated.]

If you use a pooled 2-sample t test when the two populations have substantially different variances, especially if the sample size is smaller for the group with the larger variance, you are likely to get 'false rejections' even when the two population means are equal.

In the following example, sample 1 with $n_1=15$ is from $\mathsf{Norm}(\mu_1 = 100, \sigma_1=20)$ and sample 2 with $n_2=50$ is from $\mathsf{Norm}(\mu_2 = 100, \sigma_1=5).$ And we are testing $H_0: \mu_1 = \mu_2$ vs. $H_a: \mu_1 \ne \mu_2$ at the 5% level. But a pooled t test falsely rejects $H_0$ with a P-value below 1%.

set.seed(4)
x1 = rnorm(15, 100, 20) 
x2 = rnorm(50, 100, 5)
t.test(x1, x2, var.eq=T)$p.val  # pooled
[1] 0.004382463

Of course, the 5% significance level anticipates that this kind of error will happen about 5% of the time. But when variances are unequal as in this example, false rejection or 'false discovery' happens very much more frequently than anticipated. [See Note (2) at end.]

By contrast, the Welch 2-sample t test does not require the two groups to have the same variance. If we use the same data as before, we see that the Welch 2-sample t test does not reject $H_0$ at the 5% level.

t.test(x1, x2)$p.val   # Welch
[1] 0.09780939

It is considered good statistical practice always to use the Welch 2-sample t test in preference to the pooed test, unless you have reliable advance knowledge that the two populations have essentially the same variance. In R, and many other statistical software programs, the Welch test is the 'default', so you have to so something extra to tell the software to use the pooled test. [In R, the extra step is to include the parameter var.eq=T in the t.test procedure.]


Notes:

(1) If the two population means are truly different, the Welch t test has about the same power (probability of rejecting when $H_0$ is false) as does the pooled t test. So there is seldom any downside to using the Welch t test.

set.seed(5)
x1 = rnorm(15, 95, 20)         # Null hypothesis false
x2 = rnorm(50, 110, 5)
t.test(x1, x2)$p.val
[1] 0.001372652                # Welch t rejects

(2) A simulation shows that when sample sizes and population variances are as in the beginning examples and $H_0$ is true, then the pooled test (falsely) rejects about 25% of the time (instead of expected 5%).

pv = replicate(10^6, t.test(rnorm(15,100,20), 
                            rnorm(50,100,5),var.eq=T)$p.val)
mean(pv <= 0.05)
[1] 0.252237

However, in these same circumstances, the Welch test rejects 5% of the time at the 5% level, as it should.

set.seed(2020)
pv = replicate(10^6, t.test(rnorm(15,100,20), rnorm(50,100,5))$p.val)
mean(pv <= 0.05)
[1] 0.050142

(3) Both t tests assume that we have two independent random samples and that data are normal. Both tests are somewhat robust if data are not exactly normally distributed. The major difference between the pooled and Welch tests is that the Welch test behaves as it should even when population variances are unequal.

$\endgroup$
0
$\begingroup$

I recommend using the data = ... argument inside of the t.test() function to avoid redundancies in your code. For example, this saves you extra keystrokes:

# --- Welch t procedure

t.test(pv1 ~ IND, data = data)

# --- Pooled-variance t procedure

t.test(pv1 ~ IND, var.equal = TRUE, data = data)

Your first run does not specify the argument var.equal = ... inside of the t.test() function. As a result, it will default to the Welch-Satterthwaite approximation to the degrees of freedom. This is a degrees of freedom adjustment when the variances are not equal. Your second run, however, explicitly includes var.equal = TRUE, which uses the pooled-variance $t$ procedure.

We usually make the following assumptions when performing a two-sample $t$-test:

  • Independent random samples (or a randomized experiment)
  • Normally distributed populations (though not very important with large sample sizes)
  • Equal population variances: $\sigma^{2}_{1} = \sigma^{2}_{2}$

The Welch $t$ procedure is very similar to the pooled-variance $t$ procedure. But unlike the pooled-variance $t$ procedure, the Welch $t$ procedure does not require the assumption of equal population variances. As a consequence, Welch procedures have a different standard error and degrees of freedom than the pooled-variance $t$ procedure. Because the Welch (unpooled variance) $t$-test does not make assumptions with respect to equality of the variances, it can be used in a wider variety of situations.

$\endgroup$
2
  • $\begingroup$ My output for the Welch two sample t-test says that df=307.74. Why is it not 150 since the smaller population is 151? I though it is df=min{n1,n2}. $\endgroup$
    – user283542
    Aug 2, 2020 at 11:58
  • $\begingroup$ The Welch procedure does not simply take the minimum of the two groups. Was that referenced somewhere? $\endgroup$ Aug 2, 2020 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.