0
$\begingroup$

I am analyzing exam data collected from participants who applied to an academic program. The data was collected from 2014-2018, and applicants only took the exam once when applying to the program (after being rejected, they could not reapply). The exam's questions were changed in 2018, so the participants who applied to the program during 2014-2017 took a different exam than those who applied in 2018. Both exams were validated and measure the same construct.

To examine whether race, gender, family income, and parents' marital status were predictors of passing the exam (outcome is binary pass/fail), I ran a logistic regression model on the data collected from 2014-2017 separately from the 2018 data to account for the change in the exam. However, the 2018 data has low statistical power due to a small sample size, so my model has poor fit when running it exclusively with the 2018 data. Would it be methodologically sound for me to run the model on on all of the exam data (2014-2018) controlling for the exam change in the model (0 = original exam, 1 = updated exam), or should I use an alternative method?

$\endgroup$
2
  • $\begingroup$ Someone more familiar with measurement might be better to answer this, but my instincts say yes, but only under the assumption that the exam measure the same constructs and the new exam was not changed in response to the old one being deficient in some way. Happy to be proven wrong though $\endgroup$ – Demetri Pananos Jan 7 at 19:34
  • $\begingroup$ @demetriPananos Thank you! The exam wasn't deficient but it was changed in an effort to improve inclusivity using more gender neutral items. $\endgroup$ – glen_c Jan 7 at 19:47
1
$\begingroup$

Taking from the comments the information that "the exam wasn't deficient but it was changed in an effort to improve inclusivity using more gender neutral items", there could be an interaction between the exam change variable and one or more of the other explanatory variables. This can be included in the model, then accounting for all data. In fact, in a first attempt, I'd probably even use a year indicator and interactions between this and the explanatory variables. Then one can see (maybe also using graphical methods), whether there is any clear difference between the years 2014-2017 (I wouldn't exlude the possibility a priori), and clear interactions between the year and anything, other than just at the exam change time point. The model can then probably be simplified, seeing whether modelling at year level is required or rather only the exam change, and which of these interactions is really needed. A suitably simplified model may give you a better power than analysing 2018 in an isolated manner.

$\endgroup$
1
  • $\begingroup$ Thank you--that is very helpful! $\endgroup$ – glen_c Jan 8 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.