I have two samples, one which has 1436 observations where sd=0.0405, mean=0.7776 and skewness=0.032 and the other which has 4956 observations and sd=0.0416, mean=0.7716 and skewness=-0.0897.

Now i am doing a Welch Two Sample t-test in R, but I am wondering whether I should correct for sample size, as my analysis is overpowered now?


I also am looking at effect sizes now, which seem to be nearly 0 for the initial data. This means the tests can be significant, but the result is meaningless. It also has a power of 0.93.

I decided to sample my data (with replacement) and taking only 500 observations from the 2 groups. Now I am getting an effect size of 0.152 and a power of 0.67. Which seems a better result to me.

Also my thesis supervisor suggested that I should correct for multiple testing, but I am still figuring what it exactly is and how I should perform it.

Does anyone have more suggestions on this?

  • $\begingroup$ t-test doesn't require sample size be identical. You should use two samples t-test equal variance. $\endgroup$
    – SmallChess
    Jan 19 '17 at 10:25
  • $\begingroup$ I have got a feeling that with your large data set, you will always get significant results anyway. $\endgroup$
    – SmallChess
    Jan 19 '17 at 10:27
  • $\begingroup$ That's my problem, is that a true significance then? I am now sampling my dataset with taking 500 samples (with replacement) and doing a t-test. $\endgroup$
    – Meeldurb
    Jan 23 '17 at 9:04

You are right to be aware of too much power in cases like yours. Usually, samples are scarce and expensive so one can never have enough power. In your case, however, too much power means to find alternatives that are so close to the hypothesis that we simply don't want to find them since they are practically irrelevant.

This gives some hint to a solution. It is called "test of relevance". For a test of relevance one "inflates" the hypothesis from a point to an interval of irrelevance. Now one calculates an usual confidence interval (just take the default t.test in R) and compares if this confidence interval is disjoint from the interval of irrelevance. If yes, you may reject your irrelevance hypothesis. If not --especially if the point estimation itself lies within the interval of irrelevance-- you can't infer anything.

How to choose the irrelevance margin is a matter of discussion with the practitioner. It is similar to choosing the equivalence margin for equivalence tests. Which size of a difference does he find negligible? Sometimes this is hard to answer.

Note that you may also check if an equivalence test is in fact the right test for you. If you want to show that there is at most a tiny difference, an equivalence test is your way to go. If you still want to reject the hypothesis but also want to keep some scientific diligence and fairness in face of your huge sample sizes, do a test of relevance.

Inequality of variances or sample sizes is no particular problem here. You are already treating it correctly.

  • $\begingroup$ Actually it is a test whether there exists a significant difference between the Codon Adaptation Index of duplicated and unique protein domains of an organism. That is why my sample size is so large. I have also no clue to which observations would be considered irrelevant in this kind of analysis. $\endgroup$
    – Meeldurb
    Jan 19 '17 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.