2
$\begingroup$

I've made a little questionnaire where participants can rate an answer between 1 and 5. I calculated the mean value, the average value and the standard deviation.

Now I was asking myself if it is possible to calculated a confidence interval for these results and if yes, if this would tell me anything. So I just tested it and used excel to calculate a 95% confidence interval.

Here are the values:

Arithmetic average: 4.60
Median: 5.00
Max: 5.00
Min: 3.00
Standard deviation: 0.63
95% Confidence interval: 0.32

But what is this value telling me? I can be sure by 32% that the values aren't random values? Or is a confidence interval for those kinds of questions useless?

Improve this question
$\endgroup$
4
  • $\begingroup$ Your Confidence Interval (I assume it is for the mean) looks strange because it must be an interval, not a single value. Isn't it [4.28;4.92] instead? You'll also need the sample size to compute it by hand. $\endgroup$
    – chl
    Commented Mar 26, 2011 at 15:58
  • $\begingroup$ @chl: Ah yes it is the interval... So I should use +/- 0.32. For the calculation of the confidence interval I used the standard deviation. $\endgroup$
    – RoflcoptrException
    Commented Mar 26, 2011 at 16:02
  • $\begingroup$ What is the problem you are trying to solve? What meaning does the average have for this problem? Also, is this a confidence interval for a single answer aggregated over people, or is it the mean response of the whole questionnaire by somehow aggregating the responses? Or, is it the analysis of a single questionnaire? $\endgroup$
    – cardinal
    Commented Mar 26, 2011 at 20:44
  • $\begingroup$ I would adopt a different approach here. Instead of finding "an answer" and then trying to figure out what the question is, you should start by asking a question, and then finding the answer to that question. The CI is an answer to a question, but who cares if it isn't a question you are interested in answering? So my question to you is what do you want to infer from the results of your questionnaire? $\endgroup$
    – probabilityislogic
    Commented Jun 13, 2011 at 5:05

2 Answers 2

Reset to default
4
$\begingroup$

Based on my calculations, it seems that you had about 16 or 17 participants. The typical methods for calculating confidence intervals of means assume that sample means are normally distributed. In the case of very skewed distributions, that assumption is only valid for large samples, which rules of thumb define as at least 20 or 30 (depending on whose thumb you talk to).

Also, your data are ordinal data but not necessarily interval data; the difference between a 3 and a 4 is not necessarily the same as the difference between a 1 and a 2. This also makes the typical methods less valid.

If you want to develop some quantitative measure of variability, I suggest that you use a binomial test to estimate a confidence interval of the median. But I probably wouldn't even do that; the exact numbers aren't particularly meaningful unless you have a random sample and you've tested the questionnaire for validity and reliability or if you're comparing it to a similar question or by some blocking factor.

Considering all of this, I don't trust statistics to be particularly meaningful on individual questions from questionnaires like this. When I run questionnaires like this, I generally just plot histograms. I think they tell you more than the numbers.

Improve this answer
$\endgroup$
3
$\begingroup$

IME, the confidence interval is useful as it gives a sense of how uncertain your estimates are. Its a good way to check how variable your results might be, and to give others a sense for how likely the results are to be within a particular range.

That being said, the typical interpretation of one is that 95 (for a 95% interval) times of 100 if this experiment were repeated, the true value of the mean (or whatever you've calculated the interval for) would lie in this range. So it does not tell you that it is 95% certain that the true value lies within that range. On the other hand, a bayesian credible interval will tell you this, but these are not as widely used.

Improve this answer
$\endgroup$
7
  • $\begingroup$ So basically the smaller the interval the better? $\endgroup$
    – RoflcoptrException
    Commented Mar 26, 2011 at 19:01
  • $\begingroup$ @Roflcoptr essentially, yes. However, its important to assume that the assumptions for calculating the confidence interval are met, or you could run into trouble. $\endgroup$
    – richiemorrisroe
    Commented Mar 26, 2011 at 19:06
  • $\begingroup$ @Roflcoptr Be aware that reporting only mean $\pm$ SE or a 95% CI for Likert items, whose distribution are likely to be asymmetric (and this is your case), might be misleading. Another solution would be to report mean $\pm$ SD and the frequency of observed response in the two upper responses modalities (4+5, or "agree" and "strongly agree"). $\endgroup$
    – chl
    Commented Mar 26, 2011 at 19:40
  • $\begingroup$ Your interpretation of a 95% confidence interval is incorrect. A 95% confidence interval is a set of plausible true values for the parameter given your data. It is the set of hypotheses that, when tested, are not significant at the two-sided 5% level. These hypotheses for the parameter do the best job explaining your observed data. $\endgroup$
    – Geoffrey Johnson
    Commented Nov 28, 2021 at 1:35
  • $\begingroup$ Hey @GeoffreyJohnson, not sure I see this distinction, can you clarify? $\endgroup$
    – richiemorrisroe
    Commented Nov 28, 2021 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.