I have used two different designs for a questionnaire to measure the same variable, and I want to investigate if changing the design led to a different measurement as a result.

I have a within-subject repeated measure, $n=450$ participants. Because the sample size is large, I have a significant ANOVA for the difference between the two measurements with effect size 0.004 which is very small as expected.

In another design, If I get insignificant results $p>0.05$ for two groups (using different designs), can this prove that the two measurements are equal? If not, what is the correct statistical measure to show the equivalence of the two conditions? Correlation for this group is 0.9 but I know from the literature that correlation alone is not sufficient to prove the equivalence of the two measurements.

  • $\begingroup$ In your 2nd expt, if you have 2 different methods they are likely at least a little bit different. If you used sufficiently large samples, you might be able to detect that difference---even if it is too small to be of practical interest. You already have one expt with a 'very small' effect size. You should consider what value there may be in pursuing yet another small difference. $\endgroup$
    – BruceET
    Commented May 21, 2020 at 0:38
  • $\begingroup$ The experiment is done already and I have the data. Since the slightest difference would detect a significant difference with even a small effect with this sample size, is no significant effect an indicator of no difference? What is the right statistical approach to show the equivalence? $\endgroup$
    – Artemis
    Commented May 21, 2020 at 11:28
  • $\begingroup$ It is a matter of the power of the test to detect a difference (by rejecting $H_0).$ It is not possible to "prove" that two methods of measuring are exactly the same.// Power depends of sample size. Your expt may not have enough subjects to find a difference. But a huge study with 10 or 100 times as many subjects (if someone had resources and motivation to do it) might find a tiny, but highly significant difference. // See continued comment with power curves. $\endgroup$
    – BruceET
    Commented May 21, 2020 at 18:49

1 Answer 1


About effect size, power, and sample size. The following information about the effect of sample size on power is from a recent release of Minitab statistical software.

Suppose two groups have standard deviations of $\sigma = 1$ and differ by $\mu_2 - \mu_1 = \delta = 0.1.$ That is the [effect size] $\delta/\sigma = 0.1,$ (sometimes estimated by Cohen's $d).$ Such an effect side is considered 'very small' or 'small'.

  • Then samples of size $n_1=n_2=100$ have only about one chance in ten of detecting that small difference.
  • With $1000$ subjects in each group, there is about a 60% chance of detecting the difference.

  • But with 10,000 subjects in each group, one is essentially certain to detect the difference--even though that difference may not be of practical importance. (A result to warm a P-hacker's heart.)

Power and Sample Size 

2-Sample t Test

Testing mean 1 = mean 2 (versus ≠)
Calculating power for mean 1 = mean 2 + difference
α = 0.05  Assumed standard deviation = 1

Difference    Size    Power
       0.1     100  0.10837
       0.1    1000  0.60837
       0.1   10000  1.00000

The sample size is for each group.

enter image description here

Here is an example, using simulated data and a Welch 2-sample t test. The null hypothesis is rejected at any reasonable significance level with P-value very nearly $0.$ But Cohen's effect size is $d \approx 0.3.$

x1 = rnorm(10000, 100, 1)
x2 = rnorm(10000, 100.1, 1)
t.test(x1, x2, var.eq = T)

         Two Sample t-test

data:  x1 and x2
t = -7.0915, df = 19998, p-value = 1.371e-12
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -0.12868927 -0.07295496
sample estimates:
mean of x mean of y 
 99.98982 100.09065

# Cohen's d:
[1] 0.02957828

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.