2
$\begingroup$

I have conducted an empirical study as a part of my master thesis. Each test subject filled out a questionnaire where some questions were regarding usability of different features. What is the best/correct way to calculate if a feature got a significant better usability score than another feature? (I used a 10-point Likert scale).

I had 14 test subjects, and I want to compare the usability score for a feature called "bubble" and a feature called "hull".

This is the scores:

Bubble: 6,  3,  7,      9, 7, 8, 7,  9, 8, 8, 6, 10, 3, 7
Hull:   9, 10, 10, 8.6154, 9, 6, 8, 10, 8, 8, 6, 10, 9, 9

(One subject did not rate "hull", I have therefore inserted the mean (8.6154) in its place.)

I have run a paired-sample t test which indicates that there is a significant difference in the usability scores for bubble (M=7.0, SD=2.04) and hull (M=8.62, SD=1.33); t(13)=2.43, p=0.0305.

Is this the correct way to do it? My statistical knowledge is quite limited, but I think I should say if there was a significant difference or not.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I don't see anything wrong about your work. I am not very comfortable about assuming the missing value as the mean, because you are practically changing 5% of your data (It might be alright, I am just pointing out it might be risky). Another note that is worth mentioning is that your test is not "absolutely" significant, you have to mention it was significant at 5% only, because the p value is less than 0.05. $\endgroup$
    – Error404
    Commented Aug 9, 2013 at 10:03
  • $\begingroup$ Thank you! Regarding "absolutely" significant. Is it enough to change it to say p<0.05, and by that showing I know it's not absolutely significant? $\endgroup$
    – swenedo
    Commented Aug 9, 2013 at 11:19
  • $\begingroup$ @swenedo I agree with the answer, but have a concern with the use of the Likert name for what is really a rating scale. Since this is going to be your academic work and live for a long time, I would recommend the use of rating scale. All Likert scales are rating scales, but not all rating scales are Likert scales and authors seem to want to gain credibility by using the timer Likert. Likert is a specific type of rating scale. Picky I know, but this is academia. $\endgroup$
    – doug.numbers
    Commented Aug 9, 2013 at 11:38
  • $\begingroup$ Okay, I will avoid calling them "Likert" but just "rating scale". Thanks! $\endgroup$
    – swenedo
    Commented Aug 9, 2013 at 12:27
  • $\begingroup$ I don't think it's right to simply replace the missing value by the mean of the other values -- your statistic no longer has the desired distribution. $\endgroup$
    – Glen_b
    Commented Aug 9, 2013 at 23:40

1 Answer 1

Reset to default
2
$\begingroup$

I agree with @Error404 about the assumption, esp. because the one other person who rated bubble '9' gave hull a 10. If you wanted to get fancy you could do multiple imputation; if you want to keep it simple, just drop that person.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you @Error404 and Peter. I read about single imputation at wikipedia and found replacing a missing value with the mean as the simplest method, but also noted the risk. However, I believe you are right - it ruins the correlation between two variables that might covary. When dropping that person the p-value is still less than 0.05 so I will just go with that and explain that I had a missing score which explains the lower df. $\endgroup$
    – swenedo
    Commented Aug 9, 2013 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.