We want to find the margin of error, at a 95% confidence interval on a particular question in a survey. If there are 10 questions in the survey, this question is towards the very end. Problem is, not everybody finishes the survey, so not everybody answers the question. I'm trying to find the margin of error for the results we got back on the last question.

We know how many people use Product A, let's say its 1,000 (is this the population?). The final question asks about how they would rate the product from 0 to 10. We sent out the survey to 300 people (this is the sample I'm assuming). Of the 300, only 200 complete the survey up until the last question. We want to know how reliable the results of the last question are. How will I then calculate the margin of error?

My coworker just used this: https://www.surveymonkey.com/mp/margin-of-error-calculator/

And treated the population as 1,000 and 200 as the sample, but I'm not sure if this is the correct way to go about it.

  • 1
    $\begingroup$ What is your population of interest? All people who used Product A? You mentioned that 1,000 people use Product A, but are you interested in making inferences only to those 1,000 people or to some larger population? Also, can you tell us how you sampled? Did you just randomly select from the 1,000 people? It would be good too if you could clean up your post a bit by clicking the edit button and modifying it - there are a number of typos and sentence problems that make it difficult to read and understand. Thanks and welcome to Cross Validated! $\endgroup$ – StatsStudent Jan 9 at 16:32
  • 1
    $\begingroup$ Also, you didn't tell us what statistic you were interested in calculating the margin of error on. Are you interested in the mean? $\endgroup$ – StatsStudent Jan 9 at 16:44
  • $\begingroup$ Yes, all the people who used Product A. Inferences to only people that use Product A, don't care about population outside of it. There was no random sampling? Here's how it works: we implement product A, and then we survey people about it right after. If there are 1,000 users, technically it's still growing as the number of users of product A increases. $\endgroup$ – user233540 Jan 9 at 16:58
  • $\begingroup$ Apologies, I don't even know if I'm asking the question correctly or if I put it the right way. It's been years since I've taken statistics. I'm just trying to see if the number of results we got back is statistically valid. $\endgroup$ – user233540 Jan 9 at 17:03
  • $\begingroup$ I guess an additional problem I see is that, while we have 1,000 users of Product A, we are only surveying new users of product A as they start using it. We are basing how the 1,000 and growing population of users of Product A feel about the product, based on the results from just new users who are being surveyed. $\endgroup$ – user233540 Jan 9 at 17:09

Based on the clarified comments, it is my understanding that:

  1. You have a population of people who used Product A. There are 1,000 of these people. You are interested in making inferences about these people from Q10 on your survey.
  2. You began surveying every person who purchased Product A immediately after their purchase. This began happening after 700 people had already purchased Product A, so only 300 people actually had a chance to respond to the survey.
  3. Of those 300, only 200 actually responded to the final question of the survey - the question of interest.

There are a few issues here. First and foremost, is that, since 700 people never had any chance of being surveyed, you cannot generalize your findings to these people without making what I think is a strong assumption: that these people are essentially identical in important ways to those people who were invited to complete the survey. You might not think that these people are different in any way, but one might reasonably argue that system processes or product improvements resulted in later Product A users giving higher ratings to the product or answering differently (you never told us what the question was). But since the first 700 users might have had a less enjoyable experience with Product A, but were not allowed to participate in the survey, it would not be "fair" to generalize the findings from the most recent 300 people to the first 700 people.

Next, you are not even doing any sampling. You are surveying every single individual. Since there is no sampling, there is no sampling error and hence, no margin of error. There is only potential non-response (the 100 people who haven't responded) bias. If you can assume the responses are missing completely at random, then you can simply weight your data to match the known population totals. For example, Let's say of the 200 people who actually completed the survey, 100 said "Yes" to Q10. If you are interested in approximately how many people would have said answered "Yes" out of the 300 people who should have completed Q10, then, by the simplest method (there are other more elaborate methods, but this is probably sufficient for your needs), you can simply calculate:

${300\over{200}}\times 100=150$

So you would expect, 50 of the non-respondents would have answered "Yes" and along with the 100 who did say "Yes", you would expect "150" in total would have said "Yes."

Lastly, you should be careful about describing your population of interest if a person can purchase Product A multiple times. Presumably, a person purchasing Product A will be offered a survey at the end of each purchase. If this is the case, then you should not be talking about people, but about purchases in terms of your population, without making adjustments for the possibility of multiple purchases.

  • $\begingroup$ So the last questions is actually: "Based on your experience implementing this product, how likely are you to recommend our product to friends and family?". So its a Net Promoter Score question, answers can range from 0 to 10. Let's ignore the 700+ people then, can I make an inference from the results I've gotten back so far? Can I use the 300 results I have gotten back over the last 6 months? Or am I still not able to sample? $\endgroup$ – user233540 Jan 9 at 18:10
  • $\begingroup$ The product can't be purchased multiple times. At least not in a long time period, its software with 5 to 10 year licences. $\endgroup$ – user233540 Jan 9 at 18:29
  • $\begingroup$ You can use the results, but there is nothing to sample: you've asked every single person the question. What I would simply do is simply calculate the average response of the responses you received so far and calculate a population standard deviation= $\sum_{i=1}^{200}(x_i-\bar{x})^2/200$, where $\bar{x}$ is your mean response to Q10, to get am understanding of how much your responses vary. If you think the 200 responding are similar to the 100 who didn't respond, then that IS your population statistic and hence there is no uncertainty around it's estimation. $\endgroup$ – StatsStudent Jan 9 at 18:34
  • $\begingroup$ Can I ask one more clarifying question? If I sent out a survey to 1,000 users of Product A, just once, not on a rolling basis, to get their feel for product A. And we get 400 results back, can I say the population is 1,000 and the sample size is 400? $\endgroup$ – user233540 Jan 9 at 19:33
  • $\begingroup$ @user233540. Well in that case, the population would be 1,000 and the sample size would be 1,000. So this wouldn't be a SAMPLE survey, but a complete CENSUS. But only 400 responded. $\endgroup$ – StatsStudent Jan 9 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.