2
$\begingroup$

I've made a little questionnaire where participants can rate an answer between 1 and 5. I calculated the mean value, the average value and the standard deviation.

Now I was asking myself if it is possible to calculated a confidence interval for these results and if yes, if this would tell me anything. So I just tested it and used excel to calculate a 95% confidence interval.

Here are the values:

Arithmetic average: 4.60
Median: 5.00
Max: 5.00
Min: 3.00
Standard deviation: 0.63
95% Confidence interval: 0.32

But what is this value telling me? I can be sure by 32% that the values aren't random values? Or is a confidence interval for those kinds of questions useless?

$\endgroup$
  • $\begingroup$ Your Confidence Interval (I assume it is for the mean) looks strange because it must be an interval, not a single value. Isn't it [4.28;4.92] instead? You'll also need the sample size to compute it by hand. $\endgroup$ – chl Mar 26 '11 at 15:58
  • $\begingroup$ @chl: Ah yes it is the interval... So I should use +/- 0.32. For the calculation of the confidence interval I used the standard deviation. $\endgroup$ – RoflcoptrException Mar 26 '11 at 16:02
  • $\begingroup$ What is the problem you are trying to solve? What meaning does the average have for this problem? Also, is this a confidence interval for a single answer aggregated over people, or is it the mean response of the whole questionnaire by somehow aggregating the responses? Or, is it the analysis of a single questionnaire? $\endgroup$ – cardinal Mar 26 '11 at 20:44
  • $\begingroup$ I would adopt a different approach here. Instead of finding "an answer" and then trying to figure out what the question is, you should start by asking a question, and then finding the answer to that question. The CI is an answer to a question, but who cares if it isn't a question you are interested in answering? So my question to you is what do you want to infer from the results of your questionnaire? $\endgroup$ – probabilityislogic Jun 13 '11 at 5:05
3
$\begingroup$

IME, the confidence interval is useful as it gives a sense of how uncertain your estimates are. Its a good way to check how variable your results might be, and to give others a sense for how likely the results are to be within a particular range.

That being said, the typical interpretation of one is that 95 (for a 95% interval) times of 100 if this experiment were repeated, the true value of the mean (or whatever you've calculated the interval for) would lie in this range. So it does not tell you that it is 95% certain that the true value lies within that range. On the other hand, a bayesian credible interval will tell you this, but these are not as widely used.

$\endgroup$
  • $\begingroup$ So basically the smaller the interval the better? $\endgroup$ – RoflcoptrException Mar 26 '11 at 19:01
  • $\begingroup$ @Roflcoptr essentially, yes. However, its important to assume that the assumptions for calculating the confidence interval are met, or you could run into trouble. $\endgroup$ – richiemorrisroe Mar 26 '11 at 19:06
  • $\begingroup$ @Roflcoptr Be aware that reporting only mean $\pm$ SE or a 95% CI for Likert items, whose distribution are likely to be asymmetric (and this is your case), might be misleading. Another solution would be to report mean $\pm$ SD and the frequency of observed response in the two upper responses modalities (4+5, or "agree" and "strongly agree"). $\endgroup$ – chl Mar 26 '11 at 19:40
3
$\begingroup$

Based on my calculations, it seems that you had about 16 or 17 participants. The typical methods for calculating confidence intervals of means assume that sample means are normally distributed. In the case of very skewed distributions, that assumption is only valid for large samples, which rules of thumb define as at least 20 or 30 (depending on whose thumb you talk to).

Also, your data are ordinal data but not necessarily interval data; the difference between a 3 and a 4 is not necessarily the same as the difference between a 1 and a 2. This also makes the typical methods less valid.

If you want to develop some quantitative measure of variability, I suggest that you use a binomial test to estimate a confidence interval of the median. But I probably wouldn't even do that; the exact numbers aren't particularly meaningful unless you have a random sample and you've tested the questionnaire for validity and reliability or if you're comparing it to a similar question or by some blocking factor.

Considering all of this, I don't trust statistics to be particularly meaningful on individual questions from questionnaires like this. When I run questionnaires like this, I generally just plot histograms. I think they tell you more than the numbers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.