# Large difference in sample size, high power and small effect size. Is it true significance?

I have two samples, one which has 1436 observations where sd=0.0405, mean=0.7776 and skewness=0.032 and the other which has 4956 observations and sd=0.0416, mean=0.7716 and skewness=-0.0897.

Now i am doing a Welch Two Sample t-test in R, but I am wondering whether I should correct for sample size, as my analysis is overpowered now?

# EDIT

I also am looking at effect sizes now, which seem to be nearly 0 for the initial data. This means the tests can be significant, but the result is meaningless. It also has a power of 0.93.

I decided to sample my data (with replacement) and taking only 500 observations from the 2 groups. Now I am getting an effect size of 0.152 and a power of 0.67. Which seems a better result to me.

Also my thesis supervisor suggested that I should correct for multiple testing, but I am still figuring what it exactly is and how I should perform it.

Does anyone have more suggestions on this?

• t-test doesn't require sample size be identical. You should use two samples t-test equal variance. Jan 19 '17 at 10:25
• I have got a feeling that with your large data set, you will always get significant results anyway. Jan 19 '17 at 10:27
• That's my problem, is that a true significance then? I am now sampling my dataset with taking 500 samples (with replacement) and doing a t-test. Jan 23 '17 at 9:04

This gives some hint to a solution. It is called "test of relevance". For a test of relevance one "inflates" the hypothesis from a point to an interval of irrelevance. Now one calculates an usual confidence interval (just take the default t.test in R) and compares if this confidence interval is disjoint from the interval of irrelevance. If yes, you may reject your irrelevance hypothesis. If not --especially if the point estimation itself lies within the interval of irrelevance-- you can't infer anything.