2
$\begingroup$

I have carried out some research on whether awareness and knowledge of HIV had been increased on university students by giving a questionnaire to students before and after a presentation on HIV. Only 140 students completed the pre questionnaire and only 129 completed the post questionnaire after the presentation. So how is it possible to compare whether knowledge in students increased following the presentation as the numbers are unequal before and after? Could I compare the mean or mode score before and after even though the number of participants was different?

$\endgroup$
6
  • $\begingroup$ Do you have linkages? So if ID 103A had 60% on test 1 and the same student ID 103B scored 80% on the test 2, you could say there was a 20% improvement? Do the students complete the same test at both iterations or do they receive different versions of the test? $\endgroup$
    – AdamO
    Feb 14, 2014 at 16:47
  • 1
    $\begingroup$ Hi thankyou for your reply, but I did not record individual ID for each of the students so that is why i am in a bit of a dilemma as there is no way of knowing the students that completed both questionnaires or not. $\endgroup$
    – bash
    Feb 19, 2014 at 23:54
  • $\begingroup$ Students also recieved the exact same questionnaire before and after the presentation. But I am not sure whether I can say awareness increased after the presentation as there is no way of knowing whether people completed both or only one questionnaire. $\endgroup$
    – bash
    Feb 19, 2014 at 23:56
  • 1
    $\begingroup$ That's unfortunate that you don't have ID. If you did have ID, you could pair tests and calculate a paired difference in response scores. However, this is not 100% necessary. You will have to make some assumptions, but naively going about treating these data as independent and calculating differences in pre/post treating them as separate populations still gives you a valid test. You just have less power to detect if your intervention was effective. $\endgroup$
    – AdamO
    Feb 20, 2014 at 0:07
  • 1
    $\begingroup$ @AdamO I have to disagree, as bash has described, I believe, a situation in which answers were obtained just before and after a presentation, i.e., it seems most of the respondents must have been in both pre- and post- groups. If that is true, it is too far a stretch to assume there are independent groups and to analyze the data accordingly. You have a point in that normally it increases power to take dependency into account, but here to assume no dependency will fuzz over the group differences so much that it not only will affect F- and p-values but will in fact produce an invalid test. $\endgroup$
    – rolando2
    Aug 3, 2014 at 12:46

2 Answers 2

2
$\begingroup$

Few approaches were suggested to you including

  • t-test for independent samples - but your samples are not independent,
  • repeated-measures ANOVA - but you are unable to link cases,
  • mixed-effects model - the same problems like with ANOVA, plus you have only two groups and that is a small number for random effects.

The only one of the three that you actually can use is independent samples t-test since linkage between cases is not preserved. On another hand, there are three unknowns:

  • is the change in scores significant?
  • the unobserved scores,
  • the linkage between scores.

The problem is that the samples are not independent and on another hand, without linkage the dependency was not observed. Because of that you should be more cautious with possible results. I think that what I would use is actually independent samples t-test, however the one that would provide more information on uncertainty about possible results - the Bayesian implementation of it. You could use Kruschke's BEST or the implementation by Rasmus Bååth (you can download it here).

However notice also another problem: if you gave the same test to the students than their responses would not only reflect gain in knowledge but also the fact that they already have seen the questions - so it could be boring for some to answer them for second time etc. So because of all those methodological flaws I would be very cautious with the results.

$\endgroup$
1
$\begingroup$

You can use repeated-measures ANOVA and resort to listwise deletion (i.e. use only complete cases), or you can use mixed-effects models (also known as random-effects models, multilevel models) and avoid the need for listwise deletion.

Mixed-effects models handle unbalanced designs (where the sample size differs from one time point to the next), whereas repeated-measures ANOVA doesn't. I certainly recommended looking into mixed-effects models.

$\endgroup$
2
  • $\begingroup$ Thankyou I appreciate your comment, I am personally thinking the mixed-effects model might be better for my data will have a look into how to do this. $\endgroup$
    – bash
    Feb 19, 2014 at 23:57
  • $\begingroup$ Two problems with this approach: (a) you did not record case ID's, (b) you have only two groups (some say the rule of thumb is $\geq 10$ groups for mixed models). $\endgroup$
    – Tim
    Jan 23, 2015 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.