Questions on population and sample

Question

As someone new to statistics, I had a debate with my supervisor today on the definition of population and sample in the following case.

Suppose we track all new users (1,000 in total) that used an app during a period of time, and want to look at the completion rate at each milestone step. The data looks like this below:

• Started Step 1: 1,000
• Finished Step 1: 800 (completion rate = 80%)
• Started Step 2: 700
• Finished Step 2: 600 (complete rate = 85.7%)

My supervisor's opinion is that the 1,000 new users are the whole population, because they are all the new users that used the app during the target period. Their completion rates are the true rates.

On the other hand, I think these 1,000 users are only a sample, while the whole population should be all the potential new users, whose size is unknown. And, we can only use these completion rates to estimate the true rates of the unknown population.

So the 1st question is, who is correct?

Then, what makes me more confused is that my supervisor also asks me to use a sample size calculator to make sure the population size of each step is large enough, so that the completion rates are statistically significant.

So my 2nd question is, if we treat the 1000 new users as the population, is it even necessary to check the population size? Is it not what it is?

And also a 3rd question - assuming I'm correct in saying these 1000 new users are just a sample, can I loosely say that the sample size of each step is large enough, because the numbers are much larger than 30? (I read from some materials that 30 should be the minimum size by rule of thumb).

Answer 1

1st question:
If you want to inference about 1000 users basing on 700 users you should treat 1000 as whole population (for example 80% of 800 group that finished 1st step met no technical problems, so while inferencing on whole 1000 group we should apply +-2pp confidence interval). But if you would like to inference about whole process that generates the data you should use formulas for infinite population (that would be statement of type: if any possible users would use our app, what is the probability that he will experience technical problems. And in that case confidence interval would be significantly wider, as no correction for finite population should be applied). Both of this approaches, however, require assumption, that each sample was drawn at random and with equal probabilities, that, as kjetil has said, is rather unlikely. However, in business it is quite normal to use this kind of doubtful assumptions.

2nd question:
Your supervisor seems to have very vogue understanding of statistics. Firstly: you know about every individual of 1000 group if they have completed step or not, so why should we inference anything? Secondly: fraction can be statistically significantly different than other fraction or different than zero. Fraction can not be just significant. Sometimes regression coefficients or other statistics are in statistical slang called "significant", but this usually mean that they statistically significantly differ from zero. Thirdly: to calculate sample size one must refer to size of the error he can accept, so population size can be sufficient or not to achieve certain size of error of some statistic, but can not be sufficient or not to generally investigate some fraction. So I would say, that this makes no sense to calculate required population size, however first reason is most important (if you would formulate hypothesis and required error size other reasons would not apply).

3rd question: I have mentioned it before: you can say that population size is sufficient or not for a defined hypothesis and error size you can accept on some confidence level (usually 0.95). For some very big size of error and low confidence level 30 can be enough, but in case of fraction variable it is required to get 1000 observations to get about +-3.1pp error size.

Answer 2

It really depends on what is the goals for inference. If your interest has to do with the future market potential for the app, you certainly should be interested in possible future users. That answers your question 1 for that case. Now there is a problem: Can you really think that your sample of 1000 first users is a random sample from the population of potential users? Well, the sample is certainly not random, as you cannot sample 1000 people from the population at large and force them to try the new app. Just possibly, first adopters might be more interested in the app than possible future users.

Your 2 Q I do not really understand. If the 1000 is the population, there is no inference to do.

As for the third question, no, there is no rule that states that unequivocally, $n=30$ is always "enough". Enough for what? If you want a confidence interval for the proportion of first adapters that complete stage one, then probably, for most purposes 1000 is enough.