3
$\begingroup$

I'm wondering how the following two approaches to determining the amount of data required for ascertaining statistical significance are related.

Assume I know the ratio of the first and second moments of a data sample, and that the population of sample means looks sufficiently Normal such that a $t$-test is valid.

The first approach relates to backing out the amount of data required for a one sample $t$-test to be significant at a given confidence level, when comparing the sample mean to 0.

$$ P[t > \alpha] = P[\frac{\bar{x} - 0}{s/\sqrt{n}} > \alpha] $$

Since this is a $t$ distribution with $n-1$ degrees of freedom, we could solve this by solving for $n$ and $\alpha$ iteratively, given some confidence level,

e.g. for a confidence level of 95% and a ratio $\bar{x}/s = 0.5$

> (alpha <- qt(0.95, 10))
[1] 1.812461
> (n <- (alpha/0.5)^2)
[1] 13.14006

> (alpha <- qt(0.95, n-1))
[1] 1.780576
> (n <- (alpha/0.5)^2)
[1] 12.6818

> (alpha <- qt(0.95, n-1))
[1] 1.786341
> (n <- (alpha/0.5)^2)
[1] 12.76406

Therefore in this case, to reject the null that the mean is 0 at the 95% confidence level, we would need 13 observations.

The second approach, discussed here, looks at the confidence interval around the estimate.

$$ \Delta = (\bar{x} + SE\cdot\alpha) - (\bar{x} - SE\cdot\alpha) = 2\cdot SE \cdot \alpha = 2\cdot \frac{s}{\sqrt{x}} \cdot \alpha $$

rearranging we have

$$ n = (\frac{s}{\Delta})^2(2 \cdot \alpha)^2 $$


I understand that the first approach is testing whether the sample mean differs from 0, whereas the second approach pertains to a confidence interval around the estimated mean, but I'm unclear how these are related. It seems like there is some relation between these two approaches that I'm struggling to grasp.

$\endgroup$

1 Answer 1

1
$\begingroup$

What you are referring to is generally known as power analysis. Accordingly, it is a process for determining the sample size needed to detect significance for a hypothesized effect size, or vice versa. You are correct, the two approaches you mention are related. The reference below spells some of this out. You'll see that effect side measures (e.g., Cohen's d) takes both mean differences and variability into account.

power analysis

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.