I am planning a systematic review, with one section examining the following question: to what extent do two cognitive tests (A & B) assess the same cognitive construct? In both tests, children can only be classed as passing or failing (in other words they have dichotomous outcomes). The study designs that will be included will involve a sample of children being tested on both cognitive tests, and the association between children's performance on both tests will be assessed. For each study, I will have a 2x2 association matrix with the number of children passing & failing tests A & B extracted from each study.

My first question, is what effect size is best for describing the association between the two tests, and has the best properties for meta-analysis? From a brief reading, it seems to be that the log odds ratio would work best for meta-analysis, and the final estimate could be converted to a phi-coefficient or likelihood ratio for easier interpretation for readers.

Secondly, I am also interested in correcting the final effect size for measurement error, since any association between two cognitive tests will likely be attenuated by measurement error, and not reflect the true construct-level association. Correcting for measurement error is something advocated in Hunter & Schmidt's "Methods of Meta-Analysis" textbook, although i rarely see in the literature or in highly regarded Cochrane/Campbell Collaboration reviews. I anticipate that most studies will not have any information on the reliabilities for either test A or B, so I will likely have to find this information from other sources, and also run sensitivity analyses seeing how the final result would differ with different reliability estimates.

So after doing a random-effects meta-analysis of the above , without correcting for measurement error in the individual studies, I want to be able to assess to what extent the final result would be influenced by measurement error on both tests (assuming that measurement error is invariant across the individual studies). However, I have found little guidance about how to do this with dichotomous outcomes. Which measures of association and reliability do I use? Is it fine to simply correct the RE estimate for the overall effect size using the same methods as you would correct an individual study, or do I need to use something more complicated?

  • $\begingroup$ If you can derive a standard error for your measure of choice you can use that. You are not constrained to use the log odds ratio if you do not want to. $\endgroup$ – mdewey Aug 30 '16 at 20:31
  • $\begingroup$ It's not a question of which effect sizes are possible to use, but which would best suit my needs. I've read about issues with other measures, for example, the phi-coefficient and rate ratio are influenced by the marginal distributions, which would inflate heterogeneity- I just want to make sure I pick an effect size with good characteristics. $\endgroup$ – user3084100 Aug 30 '16 at 23:38
  • $\begingroup$ Are you open to using Bayesian methods here? Speaking for myself, I honestly don't think I could express a coherent approach to your problem without employing hierarchical Bayesian modeling. $\endgroup$ – David C. Norris Sep 2 '16 at 0:16
  • $\begingroup$ I have a basic idea of those methods, and I am sure I could look up those sections in my textbook again. I use RStudio for data analysis so if that is used great. $\endgroup$ – user3084100 Sep 2 '16 at 10:31
  • $\begingroup$ In principle any R package should work under RStudio so you can search the CRAN Task View for implementations in a Bayesian framework. $\endgroup$ – mdewey Sep 2 '16 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.