I'm using some pilot data to determine the necessary sample size for an upcoming study.

The primary goal of this study is to measure the correlation between two variables, A and B, and to determine the confidence interval for this correlation statistic.

What I have been doing is looking at the distribution of differences between the two variables in the pilot data, and using the standard deviation of this distribution as a basis for a simulation. Let us call this standard deviation the ERROR.

The simulation does the following:

It generates a normally weighted random value for variable A, and then adds Gaussian noise to this value to generate a value for variable B (the standard deviation of which is equal to ERROR). It repeats this N number of times, and then measures the correlation and the confidence interval for this correlation.

At the end of the simulation, I can determine the confidence intervals as a function of N.

This is all fine, I've figured out how to do this, how to use the Monte Carlo method when the distributions in question aren't parametric, and how to test for and deal with nonindependent errors.

I am, however, having a mild crisis of faith here. The sample data is the "ground truth" from which all my simulation results are derived. Now, the goal of the actual study is to determine the correlation and confidence interval of this correlation. This correlation is directly proportional to the standard deviation of the distribution of the differences between A and B. Yet, it is this very error I am using that I am using, in my simulation, to determine how many samples of A and B I'll need to achieve a given confidence interval (I'm using the sample error, which I termed ERROR above).

Something seems circular here, and I feel that the answer is very simple and obvious. Can someone guide me back to the path of light? I think it's similar to the issue of using a sample mean to determine the distribution of sample means.


Your sense of circularity might arise from the way you stated the problem: "The primary goal of this study is to measure the correlation between two variables, A and B, and to determine the confidence interval for this correlation statistic."

Although the correlation between the two variables may represent some inherent fact of nature independent of your study, the confidence interval is necessarily related to the size of your study. Think of the confidence interval as a tool that you will use in documenting the significance of the correlation, not as an end of the study in and of itself.

Go back to what you are trying to accomplish: to decide how large a study you need to give you a specified chance (power) of finding that the hypothesized correlation (of a specific magnitude) meets a test of significance. As you recognize, the best way to make this decision is to have at least a rough idea of the true correlation and its variability. Your pilot data simply give you estimates to work with in deciding on the necessary size of the study.

  • 1
    $\begingroup$ thanks for the answer. I thought about my question more carefully and I realized that I don't even need to do any simulations to estimate necessary sample size. Coupled with your post, I think my crisis is resolved. $\endgroup$ – spacediver Mar 29 '15 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.