Calculating power of a sample (need a layman's/beginner's definition)?

We have a bi-annual survey and an intended sample size of 1,300. This sample size was determined using Krejcie's means formula for finite populations, and we drew the sample using 2-stage clustering. The data collected is primarily income, the population is quite stable and known (people in a certain industry in a particular area). This is a panel survey, so the same sample of 1,300 will be surveyed every year to directly compare changes in their income.

I have been asked to calculate the power of our sample. I have a basic understanding of what power is, and that you can either calculate the sample size taking into account the power you want, or you can perform an post-hoc power analysis. I know that it is the probability that the test correctly rejects the null hypothesis (H0) when the alternative hypothesis (H1) is true. How, then, do I perform a power analysis of the sample we currently have?

It depends on what you are trying to test. But you can look at http://www.statmethods.net/stats/power.html if you are comfortable using R. You can also work backwards and use the power.prop.test() if you are looking at a difference in proportions.

Power is also defined as:

1 - P(Accept H0|Alternative is true)

Where P(Accept H0|Alternative is true) = beta

So you can use the "pwr" package in R for the difference in means. Here is a little R code and you can change it as necessary:

install.packages("pwr")
?pwr.t.test

The only thing that is tricky is the effect size "d". This is your study, but am I assuming that you are pulling data from the same group twice, so you are going to want to use a paired t-test because the samples are not independent. In addition in R you are going to want to use an alternative statement of "less" or "greater" play around with that, I get confused which one you want here. To calculate the effect size you can look at http://www.statisticslectures.com/topics/effectsizedependentsamplest/. Hope this helps.

• Because "Alternative is true" is usually a complicated situation, requiring at least one parameter for its description, considerably more explanation appears warranted.
– whuber
Jun 13, 2016 at 20:55
• Yes, I'm not following that comment. At any rate, I am trying to test that income has increased. H0: μ1 = μ2, H1: μ1 < μ2 Jun 13, 2016 at 21:10
• @406LQE did any of this help? Jun 15, 2016 at 15:35
• @AdamWarner This is a very delayed response, but I recently found out that calculating the power of our sample is not relevant because we don't have a counterfactual to compare it to and estimate the impact of the work we are doing on our current population. Yours and Upper_Case's responses do make sense, but based on recent discussions it sounds like power calculations are both not applicable to our process and outside of the scope of our budget/abilities (we would need to be able to detect changes of 10-20 cents, because we are comparing income per person per day). May 25, 2017 at 15:51

I agree with Adam Warner that the pwr package in R is probably what you want (if you use R, that is. Other software has analogues), so this is mostly an addendum. That package requires any three of sample size, effect size, significance level, and power, and uses them to calculate the value of the fourth. In your case you have a sample size of 1,300 and your desired significance level should already have been chosen. You've been asked to calculate the power, and so you'll need to choose an effect size for your test to detect as your third argument.

The power of a statistical test depends on sample size, significance level, and the magnitude of the effect you want to be able to detect. Smaller effects are more difficult to identify than larger ones, and so with the sample size and significance level fixed your test will have greater power to detect a difference of 500 dollars than a 50 dollar difference. The effect size you choose is normally based in real-world relevance (a difference of one-one-thousandth of a cent probably doesn't matter) and I can't tell you what difference in income would be meaningful in your study.

Since this is post-hoc calculating power will give you a value describing how capable your study is of detecting a "true" increase in income. You will be reporting the power of your study to identify an effect of size d (or greater) from your sample of 1,300 respondents at your chosen significance level.