# Significant difference between difference scores of 2 groups on two measures?

I am investigating a discrepancy between male and female self reports of sexual experiences. The original survey consists of a female version (asking about victimization) and a male version (asking about perpetration). Typically, when given the original version, female's reported rates of victimization are about 2/3s higher than male rates of perpetration.

I have modified the original survey (both male and female versions) in order to determine whether the wording of the modified version will have an impact on the female/victim--male/perpetrator discrepancy. One of my hypotheses is that the modified version will produce a narrower discrepancy between female reports of victimization, and male reports of perpetration.

I need to figure out what test (or series of tests) I can use to determine if there is a significant difference between the discrepancy rate of original survey, and the discrepancy rate of my modified version.

• males and females are not matched, and I have different sample sizes of males and females
• each subject was administered both versions (original and modified) of the survey, according to gender
• subjects answered the original survey first, and then were given the modified survey
• my data will be nominal -- e.g. "Yes" I've had this experience, or "no" I haven't had this experience.
• could i do chi-squared, by combining male and female "yes"s and male and female "no"s and then fill in the contingency table with combined yes, and combined no, as my rows; and for columns use survey version? Commented Jul 19, 2013 at 0:46

To me, your hypothesis sounds like an interaction effect of the two factors "version" (of questionnaire) and "sex" (of participant). As the data are dependent / repeated measures, this should be taken into account by means of a multilevel aka hierachical aka mixed model. Instead of a linear model, you should use something like a logit link (cf. logistic regression) for your binary DV. Suppose your data are in long format, i. e., two rows for every participant, the first for the old version, the second for the new version. The DV is experience of crime 0/1. Data for 3 participants could like this:

     id sex version DV
[1,]  1   1       0  1
[2,]  1   1       1  0
[3,]  2   0       0  1
[4,]  2   0       1  1
[5,]  3   0       0  1
[6,]  3   0       1  0


According to your hypothesis, you predict a cross-level interaction between the level 1-predictor "version" and the level 2-predictor "sex" (in ANOVA / repeated measures terms, this is an interaction between a within- and a between-factor). If you code your variables like this sex: female=1, male=0; version: old=1, new=0, then you probably predict a positive beta of the interaction and you could test this one-sided. Code in R using the package lme4 could look like this (could be done with any multilevel software):

m1 <- lmer(DV ~ version*sex + (1|id), data=mydata, family=binomial(link="logit"))
summary(m1)


HTH

• thanks so much for your help! is there any way to just take the difference between male and female scores on the original (e.g. # of female "yes" [minus] #of male "yes") and compare that directly with the parallel male/female difference score of the modified version? Any way to test for significant difference between difference scores? Again, thanks so much for your help!!! Commented Jul 19, 2013 at 7:49
• There are two issues: Your data are dependent (each person has two measures), and you are interested in difference scores of proportions / means. In the chi-square arena, you may have a look at the McNemar's test for nominal, matched data, but I don't know whether it can directly handle your difference scores. Commented Jul 19, 2013 at 10:48
• I really appreciate all of your help! I thought about mcnemar's, but I have unequal sample size of male and females. I have fewer males than females. My understanding was that I'd have to have equal sample sizes to use matched pairs. Would it maybe be possible to create dummy male participants (using the mean male response to preserve the correct proportion) and equalize my male and female samples so that matched pair comparison would be possible? Commented Jul 19, 2013 at 16:46