2
$\begingroup$

I did my one sample t test twice (I have two sample sizes from a survey administered (one was in one language and the other was in another language) and compared the respective means to an established mean of 60. I also did an independent samples t test to compare the sample means to each other. Note that the sample sizes were obtained by administering the survey. I am comparing my means to a mean score of 60.

One sample t tests (N = 41 and N = 12) gave me statistically different (i.e. p < .05) means respectively to the mean I am comparing my value to. The independent samples t test did not find (p > .05) statistically differences between the sample means.

the information I input into SPSS were for one sample t test was the respective sample mean, respective standard deviation, and the null value changed to the established mean I am comparing my results to.

the output for my first one sample t test power analysis was: power = .94, N = 41, standard deviation = 18.2, effect size = .57 and significance of .05

the output for my second one sample t test power analysis was: power = .85, N = 12, standard deviation = 19.3, effect size = 1.006 and significance of .05

the output for my independent samples t test power analysis was (this was inputting the respective sample sizes, the population means for both groups which I used the ones obtained from the one sample t test, standard deviation and these two values were the respective values obtained from the one sample t test for the respective sample sizes): power = .26, N1 = 41, standard deviation 1 = 18.2, N2 = 12, standard deviation 2 = 19.3, mean difference = 9.3, significance of .05

Pleas help me understand what the output obtained from power analysis means.

$\endgroup$

1 Answer 1

0
$\begingroup$

In order to determine power for a two-sided one-sample t test of $H_0: \mu = \mu_0$ against $H_a: \mu \ne \mu_0$ at the 5% significance level, one needs to specify (a) the size $n$ of the sample, (b) the population variance $\sigma,$ and (c) the size of the difference $\Delta = \mu_0 - \mu_a,$ that one wants to be able to detect.

Some intermediate-level statistics texts have a formula for the power $P(\mathrm{Reject\,} H_0 | \mu = \mu_a)$ that uses the non-central t distribution. It uses $\sigma$ and $\Delta$ in terms of $\frac{\Delta}{\sigma}.$ One can also get a very good approximation of the power by simulation. Sometimes effect size is defined as $\theta = \Delta/\sigma.$ [Also, for one-sample t tests you can find online pages with useful 'power and sample size' procedures.]

A simulation in R for $n = 41,$ $\sigma = 18.2,$ $\mu_0 = 60,$ $\theta = 0.57,$ $\Delta = \theta\sigma = (0.57)(18.2) = 10.374,$ $\alpha = 0.05$ is shown below. I get power $0.945,$ which is close to your value. With a million iterations one can expect 2 or 3 place accuracy; also your information shows 1 and 2 place accuracy, so the difference could be due to rounding.

set.seed(722)
pv = replicate(10^6, t.test(rnorm(41,70.374,18.2), mu=60)$p.val)
mean(pv <= 0.05)
mean(pv <= 0.05)
[1] 0.945299

I don't know whether you are using a pooled 2-sample t test or a Welch 2-sample t test. Because the estimated standard deviations differ only slightly with $S_1 = 18.2, S_2 = 17.3$ it may not make much difference which test you have in mind. I'll do a simulation based on the Welch test. The simulated power is about 0.29, which is close to your value from SPSS.

set.seed(2021)
pv = replicate(10^6, t.test(rnorm(41,  0,18.2),
                            rnorm(12,9.3,19.3))$p.val)
mean(pv <= 0.05)
[1] 0.290776

You will probably not find online pages with 'power and sample size' procedures for 2-sample t tests with different sample sizes in the two groups. Also, I don't know of an exact formula for Welch two-sample tests, so for unbalanced Welch two-sample tests, simulation may be the only readily available method of finding power.

Notes: (1) Ideally, power computations are done before data is collected. Often one has to guess at standard deviations based on previous work with the same methods of obtaining data. Even so, this can help to ensure that you have a reasonably good chance of finding an effect of the desired size (if it exists).

(2) I am not exactly sure what interpretation to make based on an ad hoc "achieved" power computation. If the power is large, one might be happy to know that sample sizes seem adequate to the test at hand. If the power is low, and significant no significant result is found, one might feel that the effect is real but was not found because of an under-powered design.

However, I can think of several reasons to regard with some caution a significant difference from your two-sample test based on samples of size 12 and 41. The first reason is that it may be hard to believe that a sample of size 12 is truly representative of the population from which it was drawn.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.