1
$\begingroup$

I am writing a proposal for a research project and have been asked to calculate the sample size and power for the data I will use. The problem is that this will be an analysis of two repeated surveys of a sample of a country's population (panel panel) that I don't yet have access to (1500 respondents with complete data). What would be the best technical way of explaining this in my proposal rather than just saying "I don't have the data yet to do this". I am aware that the group I am submitting this proposal to are quite shrewd so I would like to state the theoretical reasons that I need the data as clearly and correctly as possible.

$\endgroup$
4
$\begingroup$

Why would you do a sample size calculation if you already know how many you're going to sample? Or will you need a subset of the 1,500?

There are 3 variations on the sample size calculation:

  • Sample size calculation: power set, known effect, find $n$

  • Power calculation: known $n$, known effect, find power

  • Minimally detectable effect: power set, known $n$, find effect.

Each of these historically were used to justify going about research and have become a bit of a tradition rather than something which is taken seriously, unfortunately. Largely, this is because the guesswork that's involved, you have to make some assumptions based on the population (finding the $n$), or the literature (finding the effect size). Even the NIH is somewhat more drawn toward the "interestingness" of the research or the researcher more than the feasibility of the study, in my statistical perspective. People don't think of power as a continuous variable, though, just like people always seek $p < 0.05$ they also want power $> 80\%$.

When a study is going to be done one way or another, a power calculation is moot.

It sounds like you should present a minimally detectable effect. If you are sampling, say, 100 people and you want to set power to a "good" level and say, "oh we can reject a null hypothesis with probability (our good level) if the effect is this big or bigger." this is especially useful when there is no good literature out on the thing you're trying to measure.

$\endgroup$
2
  • $\begingroup$ Adam, thank you for your comments, I am a little bit confused about minimally detectable effect, as I had never heard of this before. Is there a formula I can use? Professor Google seems to be in two minds as to the best one. For my analysis, where I want to see how increases in unemployment cause the amount of reports of poor self-rated health to increase in the same population in panel data pre and post increases in unemployment levels, what data would I need to have access to already to plug into this formula? Thanks again for all your help, $\endgroup$
    – John
    Feb 9 '17 at 12:42
  • $\begingroup$ @John I'd expect to see more than one formula since not all analyses are Z-tests. How were you planning to analyze the data? Choose the formula that matches that. In the analysis you describe, it sounds like a simple test of proportions is required. To get an idea of a range of values, just look to the literature and see the proportion differences they report. I use G*Power, free, for power analyses and it walks you pretty well through the steps of power calculation. $\endgroup$
    – AdamO
    Feb 9 '17 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.