0
$\begingroup$

I am trying to find out if the means of two groups are different. I have performed a randomization test and found significant results but I would like to confirm it with some other test. I looked into the t-test, but it seems like a more limited but less computationally-intensive version of a randomization test.

Would it be of any value to perform a t-test after already having performed a randomization test? Are there any other alternatives for a test for the differences between the means of two groups?

Improve this question
$\endgroup$
1
  • $\begingroup$ In a permutation test, you can use various 'metrics' to express the differences between the two samples: pooled t statistic, Welch t statistic, difference in sample means, difference in sample trimmed mean, difference in medians, and so on. Especially if assumptions for a t test are massively violated, a permutation test will give a different--more valid--answer (about rejection) than t test. $\endgroup$
    – BruceET
    Commented Jul 5, 2019 at 1:20

1 Answer 1

Reset to default
1
$\begingroup$

Performing several tests of the same hypothesis can lead to some very sticky situations. For instance, what would you do if you did a t-test and the test failed to reject the null? Now, you have two conflicting tests and the question becomes "what do I do now?". Not to mention that you open yourself up to making a false positive with higher probability.

My advice is to think carefully about what hypothesis you want to evaluate and use a statistical test which best fits your situation. The t-test is a canonical statistical analysis for evaluating the difference in means between two independent groups. I would prioritize that over other tests (assuming your data are consistent with the assumptions for the test).

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think the data fits the assumptions for the t-test, sort of. The distributions in the two groups look fairly normal (although both have a skew of 1 and one has a kurtosis of 1.5). This is one of the reasons I thought a randomization test would fit this situation better. But I did not realize that performing two tests on the same hypotheses can be problematic. I guess I will stick with the results of the randomization test. Thank you for your help! $\endgroup$
    – morehovschi
    Commented Jul 5, 2019 at 1:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.