Calculate minimum sample size needed for statistical significance

Question

I am implementing a market survey question and would like to know the minimum sample size needed to report the results with 95% statistical significance.

Not sure if this is relevant, but here is a little more info about the survey.

• The question is a select one type question (with 4 possible responses; yes, no, sometimes, never).
• I am hoping that at least 30% respond 'yes'
• The survey will not be implemented at random (only users of the product will be sent the survey)

I used this tool to assess the minimum sample size, which suggested a minimum sample size of 385. After reading up on the topic I realize I need to add more nuance. Since I want 30% of yes responses I think the formula should be:

Cochran’s Formula

((1.96)2 (0.3) (0.3)) / (0.03)2

1. Would I need to take into account that I want to report the 'yes' response with statistical significance?
2. Is it a problem that the survey is not sent at random?
3. Would someone please explain how to assess the minimum sample size in the most simple terms possible?

Answer 1

To determine the minimum sample size needed to report the results of your survey with 95% statistical significance, you can use a sample size calculator or formula such as Cochran's formula. This formula takes into account the desired level of precision, the estimated proportion of the population that holds a particular view (in this case, 30% of respondents answering "yes"), and the level of confidence desired (in this case, 95%).

In order to use Cochran's formula, you will need to determine the margin of error you are willing to accept. The margin of error represents the amount of error or uncertainty you are willing to tolerate in your survey results. In general, the larger the margin of error, the larger the sample size you will need.

To use Cochran's formula, you will need to plug in the values for the desired margin of error, the estimated proportion of the population holding a particular view, and the level of confidence desired. The formula looks like this:

$$n = ((z^2) \times p \times (1-p)) / (\textrm{ME}^2)$$

where:

$n$ is the minimum sample size

$z$ is the z-score corresponding to the level of confidence desired (e.g., for 95% confidence, z = 1.96)

$p$ is the estimated proportion of the population holding a particular view (in this case, 0.3)

$\rm ME$ is the margin of error It's important to note that the sample size calculated using this formula is the minimum number of respondents needed in order to achieve the desired level of precision. Depending on your specific research goals and the characteristics of your population, you may need to increase the sample size in order to ensure the reliability and validity of your survey results.

It is also important to consider the fact that your survey is not being sent at random. If you are only sending the survey to users of your product, it is possible that the results may not be representative of the wider population. In this case, you may need to adjust your sample size calculation to account for the potential for non-response bias or other sources of error.

Hope this helps a little.