Surveys: Is 25% of a large user base representative?

Question

My employer is currently running a company wide survey about attitudes towards the office i.e. Sentiment. In the past, they opened the survey to all areas of the business (Let's assume 10 very different departments) and all employees within them (Assume 1000 employees in total in the entire company) The amount of employees in each department is not equal and one particular department is probably 50% of the organisations total population.

This year, the survey is only being opened to 25% of the total employee base and the selection is 'random'

Hence, I have two queries:

• If it is a truly random selection of the entire employee base, how is that a statistically valid sample assuming all those employees responded?

• If it is random on a per department level e.g. 25% of each department, how is that a valid sample considering one department is over 50% of the total population.

I would have assumed that to determine a majority sentiment in a company, one would need at least 50% of the employee base in each department to provide a true reading sentiment.

Update: The survey is not enforced. There can be no guarantee of a 100% response rate from the 25% selected. There is no incentive or punitive means if the survey is or is not filled out.

Answer 1

Think about surveys in the general population of say the US. If we need 50% of the population to determine the majority opinion we would need a sample of about 160 million, which is truly prohibitive. Even a 1% sample is extreme (about 3.2 million), and is rarely done. An important survey in the US the General Social Survey has sample sizes between 1,500 to almost 3,000. So a 25% sample is in itself no problem.

Remember that a survey is not an election or a referendum. For the latter to be legitimate every eligible person must have the opportunity to have their say. For survey the purpose is to get a good estimate of the average opinion, and you can get that with a random sample. So the company needs to decide what the purpose of the survey is: is it a way for employees to give their opinion and participate in the company, or is it a way for the managers to get information?

Both sampling designs ensure that 25% of the employees are asked. The latter ensures that smaller department are represented in the survey. If you care about standard errors then you should take the nested nature of the sampling into account, though I don't suspect that that will matter a great deal in this case.

Answer 2

By etymology "survey" (sur- from 'super', as in 'from above' and -vey from 'view') means to get an overview, not the full picture.

So long as the 25% was truly random and not i.e. self-selected (opt-in) then it quite meets the definition of the term. If the survey is optional, then the answers will be representative only of those who feel a need to answer. For instance, imagine a restaurant in which one could fill out a feedback card after dining. Even if most diners are happy, most of the feedback will be negative because the happy customers see little reason to give feedback.

Answer 3

Another point of view comes from the theory of experiment design.

Statistical power is the probability of finding an effect if it’s real (source)

Four factors affect power:

1. Size of the effect
2. Standard deviation of the characteristic
3. Bigger sample size
4. Significance level desired

Based on these elements, you can write a formal mathematical equation that relates power, sample size, effect size, standard deviation, and significance level (source)

Under a set of assumptions, you could to characterize your survey as an experiment and tap into the design of experiment framework (here there are a couple of examples). There is a number of educated guesses to be made; however, an imperfect model might be better than no model at all.

Answer 4

I sense two questions. One about the sample size (25%, why not a majority) and another about the sampling technique (is it truly random, sample 25% randomly on the entire company, sample 25% randomly in every department, or use some other distribution).

1) The sample size does not need to be a majority. The required sample size can be anything between 0 and 100% depending on the required accuracy for a given confidence or likelihood ratio.

100% certainty is never obtained (also not with a 50% or larger subset). Achieving such high accuracy is also not the point of sampling and estimating.

See more on sample sizes: https://en.wikipedia.org/wiki/Sample_size_determination

If you get the law of large numbers you may also have an intuitive idea.

The distribution of the averages of all possible subsets (and your sample will be one of them), will become smaller, and closer to the mean of the original distribution, if the size of the subset increases. If you select one person then there is some reasonable chance that you find an exception, but to find the same exception in the same direction twice becomes less likely. And so on, the larger the size of the sampled subset the smaller the chance of an exceptional subset.

Eventually the distribution of the averages of all possible subsets has a variance equal to the variance of the original set divided by $n$ the size of the subset.

Important note! Your estimate will not be dependent on the size of the population from which you sample, but on the distribution of that population.

In the case of your 500 size department. The deviation of the averages of random subsets (of size 125) will be 11 times smaller than the original deviation. Note that the error in the measurement (the deviation of the average of the randomly selected subsets), is independent of the size of the department. It could be 500, 5000, or 50000, in all cases the estimate would be unaffected as long as they have the same distribution (now a tiny department might have some strange distribution, but that starts to disappear for larger groups).

2) The sampling does not need to be fully random. You can take the demographics into account.

Eventually you would treat each department separately in this sort of analysis and correct for variations among the departments and how you have sampled in these, differently sized, departments.

In this correction there are two important differentiations. One might assume the distribution among groups as a random variable or not. If you treat it as a random variable then the analysis becomes stronger (taking out some degrees of freedom in the model) but it might be a wrong assumption if the different groups are not exchangeable as random entities with no specific effect (which seems to be your case, as I imagine that the departments have different functions and may have widely different sentiment that is not random in relationship to the department).

Answer 5

Your question is about sample size for a finite population. But the first thing you need is the sample size required in an infinite population, which can then be used to calculate the sample size for a finite population.

In a survey of an infinite population, the formula is: $n=(z^2pq)/d^2$
$n$, sample size
$z^2$, confidence level, usually 1.96
$p$, proportion of the population with a characteristic, if unknown use 0.5
$q=1-p$, proportion of the population without a characteristic
$d^2$, error level (aka margin of error), usually 3%, but 1% or 5% can be used.

Error level becomes the most important factor because the lower the level of error, the bigger the sample size required and visa versa. Therefore, the sample size for an infinite population with 3% error is: $(1.96 \times 0.5 \times 0.5)/0.03^2=1,068$. Further, the error level means that results have an error of +/-3%, in this case. This means that if 48% of people in the survey were male, then the range possible is 48% +/- 3%, or 45% to 51%.

The next step is the formula for sample size for a finite population: $m=n / (1+((n-1)/N))$
$m$, sample size for finite population
$n$, sample size for infinite population (1,068 from above)
$N$, finite population size

Using the example of $N=1,000$, the sample size required with 3% error would be $1068 / (1+((1068-1)/1000))=517$, or 51.7% of the population.

If you used 25% of the population, the error level comes out as 5.4%. This error level may be fine based on previous surveys. With surveys there is always a trade off between the level of error you are willing to accept and the costs of doing the survey.

None of this factors in the response rate (if using a simple random sample). To find out how many people need to be contacted, you divide the sample size by the expected response rate. For example, if the previous response rate was 65%, the you would need to send the survey instrument to $517/0.65=796$ people.

Things get more complex if you want to divide up the population by department (known as stratification). Basically, you need to treat each department as a separate finite population if you want the data to be accurate to each department, which may not be practical. But you could do a stratified random sample instead of a simple random sample, where 50% of the sample is randomly selected from the department with 50% of the population, and suitable percentages are randomly sampled from other departments. It will mean that your sample size will increase slightly because you need to round all decimal places up (you can't survey 0.1 of a person). However, the results should be examined at the population (company) level and not at the department level because there will not be enough responses from each department to be accurate.

Answer 6

While talking about a valid sample, the underlying notion is usually one of representation. Does the sample "represent" the population adequately? In order to obtain a representative sample, one needs to make sure that the sample size is adequate (in order to reduce the variance of the estimate), and that the sample contains members belonging to the subsets of the population exhibiting different types of the behaviour under consideration.

First, the proportion of users selected for the survey matters lesser as compared to the absolute number of users selected. The sample size required will depend on the requirement of accuracy or confidence interval in the answer given. You can read this article for further information.

You mention that the company consists of several departments. Is is probable that the departments vary in their responses to the survey? If they do (or maybe you don't know for sure), it would be a good idea to "stratify" your sample across the departments. In its simplest form, this means picking an equal proportion of people from every department. Eg: The company size is 1000, and the sample size chosen is 100. Then you would choose 50 from a department of size 500, 10 from a department of size 100, etc. This is to avoid under-representation of a particular department in any specific "random" sample.

You also mention that not everyone may respond to the survey. If you know that roughly half the people will respond, then in order to get 100 responses, you will have to send the survey to 200 people. You will have to consider the possibility that such responses may be biased. People with a particular response may be more, or less, inclined to answer.

Answer 7

If it is a truly random selection of the entire employee base, how is that a statistically valid sample assuming all those employees responded?

It is a valid sample as long as it is drawn from the population it is meant to describe. That is, if you only sample bosses, nothing can be said about the other employees; that won't happen in the setting that you have described. It may however happen due to non-response (more on that here below).

If it is random on a per department level e.g. 25% of each department, how is that a valid sample considering one department is over 50% of the total population.

This is no longer a question of sample validity but one of sampling error. Obviously, the most precise estimates would be obtained from a stratified random draw, the stratum encompassing at least the department level. In such a setting, you will have a valid sample for each department but the estimates for small departments will be generally less precise than the estimates for big departments, thanks to the higher absolute sample size for the latter. For the overall organization, the higher sample representation of bigger departments simply reflects the reality of the organization and does in no way reduce the validity of the sample.

The survey is not enforced. There can be no guarantee of a 100% response rate from the 25% selected. There is no incentive or punitive means if the survey is or is not filled out.

You won't be able to force anyone to provide a good answer but implementing a response reminder plan is a minimum. Plus, you should explain the relevance of the survey to the employees and their impact they can have on the organisation thanks to the survey: e.g. when are the results published? what are the potential actions undertaken by the organisation based on the survey? why does each answer matter?

Once data are collected, non-response is an issue that should be dealt with. Dealing with it means you should first analyse the non-response behaviour to detect any potential patterns: has no boss responded? Has a given department not responded at all? Then adopt the necessary strategy (post-strafification, reweighting, imputation, etc.).

Answer 8

I'm expanding on @ICannotFixThis 's answer with an example on how the four factors involved matter:

1. Size of the effect
2. Standard deviation of the characteristic
3. Bigger sample size
4. Significance level desired

How these factors affect your results will depend on the statistic you are using. For example, if you wanted to guess at the mean of some variable you might use Student's T Test.

Let's assume you want to figure out the average height of your employees with this survey. You don't actually know the standard deviation of the height of all employees at your company (without measuring everyone) but you could do some research and guess at 3 inches (it is roughly the standard deviation of height for males in the US).

If you surveyed only 5 people then 95% of the time the average height you observe in your survey will be within 3.72 inches of the true average height.

Now, how do our factors affect this:

1. If you need to know the average height very precisely (e.g. the effect size is very small) then you will need a large # of samples. For example, to know the true average height within 2.66 inches you would need to survey 100 people.

2. If the standard deviation is large then the precision you can obtain is going to be limited. If the standard deviation were 6 inches instead of 3 inches and you still had 5 responses you would only know within 7.44 inches instead of 3.72 inches the true average height.

3. Skipping this point since it is the focus of the entire discussion.

4. If you really need to be sure you have the correct answer then you will need to survey more people. In our example we saw that with 5 responses we could get within 3.72 inches 95% of the time. If we wanted to be sure our answer was in the correct range 99% of the time then our range will be 6.17 inches and not 3.72 inches.