I have the regular hypothesis where I'm testing whether the mean of one variable is larger than the mean of the other variable or not:

H0: mean(variable1) <= mean(variable2)
H1: mean(variable1) > mean(variable2)

I've ran the Welch's test variation of the student t test in R and my output is:

    Welch Two Sample t-test

data:  x and y 
t = -2.8207, df = 43.367, p-value = 0.9964
alternative hypothesis: true difference in means is greater than 0 
95 percent confidence interval:
 -0.08806587         Inf 
sample estimates:
mean of x mean of y 
0.6708163 0.7260000 

The significance level I'm using is 5%. So based on the output, I know I can't reject the H0 hypothesis on a 5% significance level. Since the difference in mean from x and y is about -0.05 and is within the 95% CI, I can yet again conclude that I can't reject the H0 hypothesis.

Subquestion: Does analyzing both the result with the CI and the p-value somehow empower my statement? Can I be 'more confident' that H0 can or cannot be rejected if I analyze the CI if my p-value already indicates it's above or below the significance level? I assume a contradiction between p-value testing and CI testing cannot occur.

Main question: how should I interpret the t-value exactly? From what I learned, the 'further' the t-value strays from 0, the more likely that the effect is 'statistically significant'. First of all, I'm not sure what it means exactly when they say an effect is statistically significant. I'm guessing it means our test is more likely to be accurate? And how far from 0 would be safe and why?

  • 3
    $\begingroup$ 1. What is the exact meaning of the t-value in the student t test? It's the number of standard errors of the mean difference that the means differ by. $ $ 2. What's a "BI"? Do you mean the confidence interval for the difference in population means? What's "B" then? $\endgroup$ – Glen_b Dec 8 '13 at 15:27
  • 1
    $\begingroup$ Ah.. that clears things up quite a bit. Thanks! And sorry, with BI I meant the confidence interval. It's actually the abbreviation for the Dutch term, but I was so used to using it like that that I forgot to change it to English. I'm going to adjust it. $\endgroup$ – Babyburger Dec 8 '13 at 15:50
  • 1
    $\begingroup$ If your confidence interval has the same number of sides as the test (one sided or two) in the same direction, and the interval is of coverage $1-\alpha$ for the same $\alpha$ as the significance level of the test (that is, if the two things correspond), they will usually tell you the same information in respect of the hypothesis test (in this case I think it's exactly the same information). If you use them inconsistently, then they might appear to be 'contradictory'. $\endgroup$ – Glen_b Dec 8 '13 at 15:53
  • $\begingroup$ 'statistical significance' is the circumstances that hold when H0 is rejected by a hypothesis test; that is, when the p-value is $\leq$ $\alpha$, or (equivalently) when the test statistic is at least as extreme as the critical value. $\endgroup$ – Glen_b Dec 8 '13 at 16:41
  • $\begingroup$ @Glen_b I might be misinterpreting you, but are you saying that (given comparable tails) the 95% CI boundaries correspond to an $\alpha=0.05$ rejection threshold? Because there's plenty of arguments in the "visual hypothesis testing" literature that (a) no they are not, and (b) how to either adjust CIs to support inference, or how to construct such "inferential intervals". $\endgroup$ – Alexis Jun 26 '14 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.