2
$\begingroup$

I am collecting averages of scores between 1 and 5 on a customer satisfaction survey. Sample sizes are routinely less than 20 for shorter periods. (Over longer periods, this is not a problem, as the sample size increases sufficiently.)

The population mean is expected at 4.78, and the population standard deviation is estimated to be .6.

I would have loved to use a t-table with n-1 degrees of freedom to get the confidence interval for two or three standard deviations for the sampling distribution. Unfortunately, with a sample size of 20, both two and three standard deviations to the right extend beyond the range of possible scores, meaning that sample size isn't even approximately normally distributed, right?

I am more interested in the spread of the data to the left, but I don't want this to be thrown off from the spread of the data to the right.

How can I get the probability of scoring a certain amount below or above the mean in such a situation with such a sample size?

$\endgroup$
9
  • $\begingroup$ It seems odd that you're treating these apparently ordinal scores as numerical values. With a mean of 4.78, there's zero probability that someone scores 0.75 below the mean (they can only score a 4). Similarly, the probability that someone scores at least 0.8 below the mean is identical to the probability that they score at least 1.5 below the mean (both are between 3 and 4). A normal distribution isn't the right tool to describe an ordinal scale. $\endgroup$ Commented Sep 12, 2019 at 13:22
  • $\begingroup$ @NuclearWang - It is possible to score below a 4. Possible raw scores are 1,2,3,4,5. But the scores are averaged. So if I had a sample of one 5 and one 1, the average score would be 3, 1 point below 4. It is not possible to go over 5 or below one, however. $\endgroup$
    – Mr. A
    Commented Sep 12, 2019 at 13:26
  • $\begingroup$ Since your data is discreet and you mean value is near the upper end of the allowable ranges. you may want to consider using a bootstrap sampling technique to estimate the limits instead of the t-score. $\endgroup$
    – Dave2e
    Commented Sep 12, 2019 at 13:36
  • $\begingroup$ @DaveT Tell me more? $\endgroup$
    – Mr. A
    Commented Sep 12, 2019 at 13:53
  • $\begingroup$ Aside from bootstrapping (which is nice in being very straightforward) you may also compute a distribution more directly/exactly. Or at least, for a given categorical distrubution (probabilities of the different classes/scores) you should be able to compute, with an exact function, the sample distribution of the mean. And you can estimate that distribution with your data. $\endgroup$ Commented Sep 12, 2019 at 15:34

3 Answers 3

1
$\begingroup$

From Wikipedia: "In statistics, bootstrapping is any test or metric that relies on random sampling with replacement." See the Wikipedia article for the list of advantages and disadvantages. https://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29

The basic procedure is to assume your sample of N individuals represents the distribution of the population.
Now sample your sample N times with replacement and calculate your test metric and record the results. Now repeat. The distribution of the test metric should now estimate the sample variation of the population. From estimate determine the confidence limits.

Example: You have sample of 10 people with the following scores (2, 3, 3, 3, 3, 4, 4, 4, 4, 5), mean 3.5
Now sample these scores with replacement 10 times and calculate a new mean.
Now repeat many times. Resulting in a list of values (3.7 3.6 3.6 3.2 3.7 3.5 …)
The distribution the calculated means is the distribution estimate. The histogram below is shown after 1000 resamples:
enter image description here

$\endgroup$
1
$\begingroup$

This sounds like a really good application for the multinomial distribution. Since the scores are ordinal (1 through 5), it doesn't make sense to treat them as numerical. That your proposed confidence intervals exceed the upper limit of 5 is a good sign that this is the case.

Instead, let's model the data as multinomial with 5 categories. An estimate for the multinomial parameter $\hat{\pi} = \left( \hat{\pi}_1, \hat{\pi}_2, \dots, \hat{\pi}_5 \right)$ is simply

$$ \hat{\pi}_j = \dfrac{1}{n} \sum_i \mathbb{I}(x_i=j) $$

Simply count up the number of times you observe a rating of 1, for example, and divide by the total sample size. This is your estimate for the probability that you observe a 1.

To estimate the probability we observe a score lower than some category, let's first estimate the odds of observing a category as opposed to all the others that precede it. We can use continuing ratio logins to do so

$$\hat{\theta}_j = \log\left(\dfrac{\hat{\pi}_j}{\sum_{i<j} \hat{\pi_i}}\right)$$

$\hat{\theta}_j$ is the estimate of the odds of falling in category $j$ as opposed to falling in any of the preceding $j-1$ categories. The variance of this estimator is given by

$$ \operatorname{Var}(\hat{\theta}_j) = \dfrac{1}{n} \left( \dfrac{1}{\sum_{i<j} \hat{\pi_i}} + \dfrac{1}{\hat{\pi}_j} \right) $$

This expression is found in chapter 2 of Lachin's "Biostatistical Research Methods" Second edition.

The logits are assumed to be asymptotically normal, which means we can apply the simple estimate + 1.96 the standard deviation. Then, we can transform the confidence interval back into the probability space to obtain the desired probability via an inverse logic transformation.

$\endgroup$
0
$\begingroup$

As a general statement, the $t$-test is used when you're assuming that each sample comes from a population with an unknown mean and unknown standard deviation. You should consider whether you should model the samples as varying in both their mean and their standard deviation ($t$-test), or having a fixed standard deviation and varying mean ($z$-test).

However, the above doesn't apply in the case your discussing. Both tests assume that the underlying distribution is normal. Here, you have a multinomial distribution, but it's acting much like a binomial. If the population mean is 4.78, then the majority of responses are fives. (If all of the responses are fives and fours, then 78% are fives. If some of them are smaller then four, then the percentage of fives must be even higher.) Since the results are so dominated by fives, it can for many purposes be analyzed by simply dumping all the non-fives into one bucket, giving a binomial distribution, without losing much accuracy. Binomial distributions converge to normal somewhat slowly, and they are especially slow when the single-trial probability is far from 0.5, as is the case here. You should look into, rather than treating it as normal and trying to estimate the parameters of $\mu$ and $\sigma$, treating it as binomial and trying to estimate the single-trial probability $p$ that someone will give a five. You can find more information here: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

How can I get the probability of scoring a certain amount below or above the mean in such a situation with such a sample size?

If you mean the probability of a single person giving a particular score, that's not really something that you can derive from this sort of analysis. There's no reason to think that the probabilities of the different scores follow a normal, or any other standard, distribution. You'll just have to treat this as a four unknowns that you need to estimate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.