Typical t-test checks if two sample means are significantly different from each other. Let's assume the following two sample sets:

x1 = [12.9, 10.2, 7.4, 7.0, 10.5, 11.9, 7.1, 9.9, 14.4, 11.3]
x2 = [10.2, 6.9, 10.9, 11.0, 10.1, 5.3, 7.5, 10.3, 9.2, 8.8]

Question: Is the mean of x1 significantly different from the mean of x2? (mu1 != mu2)

I can construct their 95% confidence interval of means:

stats.t.interval(1 - 0.05, len(x1) - 1, loc=np.mean(x1), scale=stats.sem(x1))
>>> (8.461873578892417, 12.058126421107586)

stats.t.interval(1 - 0.05, len(x2) - 1, loc=np.mean(x2), scale=stats.sem(x2))
>>> (7.663208497074507, 10.376791502925492)

There's an overlap between the two confidence intervals, and therefore the sample means are not significantly different. We can also formally test this using t-test assuming equal variance:

stats.ttest_ind(x1, x2, equal_var=False)
>>> Ttest_indResult(statistic=1.245268949149111, pvalue=0.23018336828903668)

Since the p-value is not smaller than 0.05, we cannot reject the null hypothesis, and conclude that sample means are not significantly different.

But what if I want to know if one group has higher means than the other?

Question: Is the mean of x1 significantly bigger than the mean of x2? (mu1 > mu2)

I've heard that there's a difference between two-tailed and one-tailed test, and I think my question is related to it. How can I tell if something is significantly BIGGER or LESS than the other? Not just something is significantly DIFFERENT from each other? How do I quantify that difference?

  • $\begingroup$ You want a $t$-test with a one-sided hypothesis and the corresponding one-sided confidence interval. Look here and here. You'll find that the corresponding $p$-values is just half of yours, namely $0.1151$. The 95%-confidence interval for the difference of means is $(-0.4938; \infty)$. $\endgroup$ – COOLSerdash Jun 30 '19 at 7:28
  • $\begingroup$ @COOLSerdash I am specifically interested in conf. int of difference in means that assumes one-tailed test. Could you provide codes that you used to come up with (−0.4938;∞)? Either R or Python is fine $\endgroup$ – Eric Kim Jun 30 '19 at 7:52
  • $\begingroup$ @COOLSerdash and one more, what does that (−0.4938;∞) tell me about my samples? How is -0.4938 related to the p-value of 0.1151? $\endgroup$ – Eric Kim Jun 30 '19 at 7:55
  • $\begingroup$ The code I used in R was just t.test(x, y, alternative = "greater", var.equal = FALSE). $\endgroup$ – COOLSerdash Jun 30 '19 at 8:10

Welch one-tailed t test. A Welch (separate variances) 2-sample t test of $H_0: \mu_1 \le \mu_2$ against $H_a: \mu_1 > \mu_2$ in R gives the following result:

x1 = c(12.9, 10.2, 7.4, 7.0, 10.5, 11.9, 7.1, 9.9, 14.4, 11.3)
x2 = c(10.2, 6.9, 10.9, 11.0, 10.1, 5.3, 7.5, 10.3, 9.2, 8.8)

t.test(x1, x2, alte="gr")      # Note param 'alte="gr"` for right tailed test

        Welch Two Sample t-test

data:  x1 and x2
t = 1.2453, df = 16.74, p-value = 0.1151
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 -0.4937982        Inf
sample estimates:
mean of x mean of y 
    10.26      9.02 

Testing with P-value: The sample mean $\bar X_1=10.62$ is not significantly larger than the sample mean $\bar X_2=9.02$ at the 5% level of significance because the P-value $0.115 > 0.05.$

For comparison: a two-tailed test of $H_0: \mu_1 = \mu_2$ against $H_a: \mu_1 \ne \mu_2$ would have a P-value about 0.23 (twice the P-value of the right-tailed test shown above). As you have seen: no significant difference.

Testing with critical value: Because the test statistic $T$ is approximately distributed according to Student's t distribution with 16.75 degrees of freedom, the critical value for a right tailed test at the 5% level is $c = 1.741,$ so that one would reject if $T > c = 1.741,$ but this is not the case. [Again for comparison, the critical value for a two-sided test would be $c = 2.112,$ with rejection for $|T| > 2.112,$ also not the case.]

qt(.95, 16.74)
[1] 1.741163
qt(.975, 16.74)
[1] 2.112315

Visualization. Boxplots of the two samples, plotted sided-by-side on the same scale, illustrate the considerable overlap of values between the two samples.

boxplot(x1, x2, col="skyblue2")

enter image description here

For both the right-sided and the two-sided test, the null distribution is Student's t distribution with 16.74 degrees of freedom; its density function is shown in both graphs below. At left, the critical value $c = 1.741$ is located at the vertical dotted line; 5% of the area under the density curve lies to the right of that line. The solid blue vertical line shows the observed value $T = 1.25$ of the test statistic; the area under the density curve to the right of this line is the P-value. At right, the critical values $\pm 2.112$ are shown as dotted vertical lines, each cutting 2.5% of the area from a tail of the distribution. Taking into account that $T = -1.25$ would have be just as extreme (in the directions of the alternative) as $T = 1.25,$ the P-value gets doubled.

enter image description here

Notes: (1) Each sample separately is consistent with random sampling from a normal population. Samples are small, but we can see that there is no marked departure from normality that would interfere with the validity of a t test. (2) The sample variances are noticeably different, but not significantly different. Lacking advance information that the two populations have essentially the same variance, I followed standard practice and used the Welch t test, which does not assume equal population variances. (3) However, the P-value of of a right-tailed, 2-sample pooled t test is also about 0.11, so it wouldn't have found a significant difference either. (4) In the Welch test, R uses fractional degrees of freedom for the t distribution. Some software and almost all printed tables would use 16 or 17 degrees of freedom instead of 16.74; results for non-rejection would be unchanged.

  • $\begingroup$ great answer. One question - do you know how the one-sided confidence interval (-0.4937982, Inf) is used to draw a statistical conclusion? It seems that conclusions are drawn from t-statistic directly, and not use the one-sided confidence interval $\endgroup$ – Eric Kim Jun 30 '19 at 20:57
  • $\begingroup$ in two-tailed test, i get this interval: (-0.8633815 3.3433815). Since the two-sided confidence interval of difference in means has 0 in it, i can conclude with 95% confidence that sample means are the same. Can I use similar concept in one-tailed test? $\endgroup$ – Eric Kim Jun 30 '19 at 20:59
  • 1
    $\begingroup$ (1) A one-sided CI essentially gives a bound, here a lower bound for the difference. Interpretation is that the CI $(-.49, \infty)$ includes $0,$ so that "no difference" is a believable value for $\mu_1 - \mu_2.$ (2) Right. (3) Fussing with an add'l figure. Possibly coming soon. $\endgroup$ – BruceET Jun 30 '19 at 21:10
  • $\begingroup$ @EricKim. You said "Since the two-sided confidence interval of difference in means has 0 in it, i can conclude with 95% confidence that sample means are the same." NO! You cannot be at all sure the means are the same. You know with 95% confidence (with some assumptions) that the difference between the means is within the range of the CI (that is a gross simplification of the definition of the CI). You know that if the means are the same, the kind of difference you saw would not be super rare. But you don't "know" that the means are the same. $\endgroup$ – Harvey Motulsky Apr 8 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.