2
$\begingroup$

I conducted a T-test to see if two independent groups (English (n=324) vs. German-speaking participants (n=346)) differ in their motivation to do activities for fun (Likert scale instrument).

I use JASP for this analysis.

Null Hypothesis: The population means are equal.

Alternative Hypothesis(1): The two population means are not equal.

Alternative Hypothesis 1

Alternative Hypothesis(2): Mean (Group 1) > Mean (Group 2) :

Alternative Hypothesis 2

Now, based on the p-value of alternative hypothesis 1, I cannot reject the null hypothesis i.e.the two means are not significantly different. This makes sense.

The effect size (Cohen's D) is 0.135.

What does not make sense to me is the p-value of the alternative hypothesis 2.

How should I interpret this? The p-value suggests I can reject the null-hypothesis?p

Usually, textbooks and online resources on independent samples t-test only discuss alternative hypothesis 1 (unequal means). Is it nonsensical to have the other alternative hypothesis here? Or should I choose a smaller p-value given the relatively large sample size?

$\endgroup$
1
  • 5
    $\begingroup$ Practically speaking, you might be putting too much emphasis on the p value. If you have over 600 observations, and a Cohen's d of 0.1, is the mean difference between these two groups of any practical importance ? ... But to get to the question, really, it's valid to use the one-sided p value only if you planned ahead of time to test if Group 1 > Group 2. $\endgroup$ Dec 2, 2021 at 11:15

1 Answer 1

3
$\begingroup$

There are a couple of things to note here.

In the case of the tests you've done the p-value of the 2-sided test will be twice that of the ">" test.

In hypothesis testing we are generally looking at values as or more extreme than the observed test statistic. For 1-sided ">" t-test we calculate the (tail) probability of getting a value greater than the t-test statistic. This is the 0.040 value. When applying the 2-sided t-test to this data we look at tail probabilities on both the upper and lower tails based on the test statistic. Treating the test stat in a positive sense (abs(t-test statitic)) we calculate the upper probability, and in a negative sense (-abs(t-test statistic)) we calculate the lower probability, then add the probabilities to get the p-value. As the t-dist. is symmetric this in effect is just double the value of the ">" case. Hence the 0.080 value. If you are using significance levels (e.g 5%) to assess the results, then in a sense, the 2-sided test has to test "more" (both sides) at the same signifiance level as the one sided test (i.e. a more extreme value of the test statisitc is needed to reach significance than for the one sided test at the same level).

Similar for other tests.

So, you are performing two similar but actually different tests on the same data. Some questions to consider are:

  1. Why are you doing this: is this a post-hoc analysis? If you had done the "<" test, what would that have told you?

  2. The tests are different so would you really expect the same results from both? Especially given how the p-values are calculated?

Additional edit: As @Frank Harrell suggests in the comment below, a non-parametric test is more appropriate (the data is on a Likert scale).

$\endgroup$
2
  • 3
    $\begingroup$ The original post smells of data dredging. Pre specify your test and hypothesis and stick to them. Hard to argue against Wilcoxon here . $\endgroup$ Dec 2, 2021 at 16:10
  • 1
    $\begingroup$ If we used Bayesian methods more often, much of this problem would vanish. With Bayes we are interested in directional evidence and would be less likely to compute a measure of bidirectional evidence. $\endgroup$ Dec 26, 2021 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.