3
$\begingroup$

Is it good practice to correct for reliability of your measure in a meta analysis? This is what is suggested by Hunter and Schmidt (2004), but it seems weird to me to calculate your average effect for a perfect situation (perfect reliability) that will never exist. It also feels like the reported effects will be larger in this type of meta analysis than is ever possible to obtain in the real world of data collection? Can you refer me to some more reading & give my your personal opinion on this procedure?

References

Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Newbury Park, CA: Sage

$\endgroup$
3
$\begingroup$

Noel Card (2011) discusses the merits/drawbacks for correcting for unreliability (and other artifacts, like artificial dichotomization) in his book. The text isn't the most sophisticated resource for meta-analytic methods--it's essentially a meta-analysis for beginners resource--but I do think it does a really good job presenting both sides for procedural decisions like the one you're thinking about (and many others).

In a nutshell, it's a decision that you need to make for yourself; some researchers support correcting for artifacts, whereas others do not for many of the reasons you offer above. For peer-review, it therefore is important for you to justify your decision--whether you correct for artifacts or not.

In my own opinion, I think there are likely occasions where artifact correction is useful, and other cases where it is less useful. For example, I am currently conducting a meta-analysis of correlations between two psychological variables, and often times they are just studied together because their connection seems very "intuitive". I have used artifact correction (for both unreliability and artificial dichotomization), and am finding that there is no significant meta-analytic correlation, despite the correction. So because I used artifact correction, I actually have a much stronger demonstration of there being no correlation, because the corrected correlations represent the strongest possible association (real-world-liness be damned) that the two variables could have. If I hadn't used artifact corrections, someone who supported the intuitive connection between the two variables might respond, "well, maybe unreliability of the measures is masking a true correlation!" And now I have direct evidence to the contrary.

Finally, you could always analyze your data both ways and present estimates from both analyses--with and without artifact correction. This is sometimes called a "sensitivity analysis" (Borenstein, Hedges, Higgins, & Rothstein, 2009). You can note then the differences, if any, between the two approaches to data analysis, and then justify one model over another, or illuminate circumstances where either model might be more useful.

References

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, UK: Wiley.

Card, N. A. (2011). Applied meta-analysis for social science research. New York, NY: Guilford Press.

$\endgroup$
0
$\begingroup$

Researchers are often interested in estimating the latent true correlation between two constructs (i.e., assuming perfect measurement). We typically measure variables with error, and the reliability of measurement can be quantified in various ways (i.e., various ways of quantifying and defining the proportion of the observed data that reflects true score variance).

The typical correction formula is described here: https://en.wikipedia.org/wiki/Correction_for_attenuation

Is it good practice to correct for reliability of your measure in a meta analysis?

In most meta-analyses in psychology examining correlations, you will see that both corrected and uncorrected correlations are reported.

This is important for a variety of reasons. For example, often you'll be pooling correlations that used different measures of a variable. In my research domain, you might have some studies that used a 4 item per scale measure with an alpha of .65 and others wich used a 20 item measure with an alpha of .90. Studies that used the more reliable measure will tend to show larger correlations with other variables just because of the measurement used. So it can be helpful to correct for this factor.

That said, corrections involve assumptions that may not be accurate. For instance, there are various measures or reliability. Estimates of reliability are measured with error themselves. Often primary studies in the meta-analysis will not report reliability and you will have to rely on other sources. The assumption of the correction formula that errors in measurement are uncorrelated may not be satisfied in practice.

In short, while the goal of correcting for reliability may be worthwhile, the achievement of that goal rests on assumptions that are most likely only approximated in practice.

Furthermore, it's much better for researchers to do research where they actually measure their variables with a high degree of reliability than measure with low reliability and rely on corrections to estimate true relationships.

It also feels like the reported effects will be larger in this type of meta analysis than is ever possible to obtain in the real world of data collection?

Effects after correcting for reliability will be larger. But that is by design. This is not a problem per se. That said, we often treat correlations as something other than what they are. I.e., we develop rules of thumbs around what is a small, medium, large correlation in our domain. Thus, you have to be careful in applying intuitions about uncorrected correlations to the interpretation of corrected correlations. Furthermore, if you are going to do a study and you are doing a power analysis or some such, then you shouldn't use unattenuated meta-analytic estimates (or at least not without attenuating them based on your expected reliabilities).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.