1
$\begingroup$

In my research (medical education field), I had 32 participants answering a questionnaire consisting of 40 Likert items rating their level of confidence in different aspects of treating a certain disease (1= Not at all confident, 6= Completely confident). The participants then undertook a course, and rated their confidence again after the course, using the same questionnaire (exactly same 40 Likert items). My hypothesis is that their self-rated confidence after the course is significantly higher than before the course.

So my question, as a total beginner in statistics, is what is the most appropriate method on comparing my sample's confidence, before and after this course. Should I calculate the mean of each participant, and then the mean of the means before and after and compare those using a paired t-test, or am I thinking completely wrong here?

$\endgroup$

2 Answers 2

2
$\begingroup$

Your approach seems reasonable. One difficulty is that interpreting the mean as a level of confidence implicitly relies on the fact that confidence varies in the same way across the board (as opposed to being specific to some items/aspects of the disease). Your analysis provides no way to check that or look at more fine-grained variation but it is certainly not completely wrong.

Apart from that, there is a whole lot of things you could do but whether it is appropriate or makes sense to spend time doing anything fancy with such a small sample size is somewhat doubtful. A common way to treat this type of data would be to use exploratory factor analysis to understand the structure of the questionnaire or confirmatory factor analysis/latent variable modeling to see if all 40 items really do reflect a single underlying construct (“confidence”) and to extract a cleaner measure of this construct. However, all this would typically require a (much) larger sample size. Still, you could at least look at the correlations between items (at each time point) and hope that they are not ridiculously low but with 30 participants you would in any case expect huge sampling variability. MANOVA might also be interesting, especially if you want to include other explanatory variables (say student characteristics).

$\endgroup$
1
  • $\begingroup$ Thank you very much. I will take in consideration all your comments. $\endgroup$
    – Ioannis
    Apr 23, 2013 at 20:25
0
$\begingroup$

Loannis,

If your research questions are: 1) are participants more confident, and 2) is the change in confidence significant, then the approach might be the following. A simple calculation of the after confidence level minus the before confidence level for each question for each participant could be calculated with a spreadsheet. You could then report the range of changes and the average change in confidence levels. The data could also be plotted as comparative histograms or comparative boxplots to visualize the change for participants.

The issue of whether the change is significant could be approached with a paired t-test. The average confidence level for each participant could be compared before and after.

Your sample size is really too small to do anything fancier; and you shouldn't use a more high powered test than you really need to answer your research question.

My response presumes that you don't want to get into validating the questionnaire, but simply want some feedback. I do have a concern with the length of the questionnaire. With 40-items you may have some survey fatigue. Checking the amount of variance for the last half of the items compared to the first half should reveal if fatigue is an issue. A quick Cronbach's Alpha estimate will also provide you some feedback on the internal consistency of the questionnaire.

$\endgroup$
1
  • $\begingroup$ Thank you so much for your reply. I will definitely consider estimating Cronbach´s alpha, especially since I consider looking at different "clusters" of components of the aforementioned questionnaire. $\endgroup$
    – Ioannis
    Apr 23, 2013 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.