11
$\begingroup$

Background:

A typical meta-analysis in psychology might seek to model the correlation between two variables X and Y. The analysis would typically involve obtaining a set of relevant correlations from the literature along with sample sizes. Formulas can then be applied to compute a weighted average correlation. Then, analyses can be performed to see whether correlations vary across studies by more than would be implied by mere effects of random sampling.

Furthermore, analyses can be made a lot more complex. Estimates can be adjusted for reliability, range restriction, and more. Correlations can be used in combination to explore meta structural equation modelling or meta regression , and so on.

However, all these analyses are performed using summary statistics (e.g., correlations, odds ratios, standardised mean differences) as the input data. This requires the use of special formulas and procedures which accept summary statistics.

Alternative approach to meta-analysis

Thus, I was thinking about an alternative approach to meta-analysis, where raw data is used as input. I.e., for a correlation the input data would be the raw data used to form the correlation. Obviously, in most meta-analyses several if not most of the actual raw data is not available. Thus, a basic procedure might look like this:

  1. Contact all published authors seeking raw data, and if provided, use actual raw data.
  2. For authors that do not provide raw data, simulate raw data so that it has identical summary statistics as those reported. Such simulations could also incorporate any knowledge gained from the raw data (e.g., if a variable is known to be skewed, etc.).

It seems to me that such an approach might have several benefits:

  • Statistical tools that use raw data as input could be used for analyses
  • By at least obtaining some actual raw data, authors of meta-analyses would be forced to consider issues related to the actual data (e.g., outliers, distributions, etc.).

Question

  • Are there any problems with performing meta-analysis studies on a combination of true raw data and data simulated to have identical summary statistics to existing published studies?
  • Would such an approach be superior to existing methods of performing meta-analyses on summary statistics?
  • Is there any existing literature discussing, advocating, or critiquing this approach?
$\endgroup$
  • 1
    $\begingroup$ Re #1: most of the time, it seems like it would be insanely difficult to coerce all the data into the same format! Re #2: it will be unless a) you use particularly bad methods, or b) the summary statistic is also a sufficient statistic for the parameters you are interested in. $\endgroup$ – Andy McKenzie Jun 12 '12 at 6:08
  • 1
    $\begingroup$ @Andy McKenzie Re #1: That's what I am currently doing (i.e. IPD meta-analysis). Given that my outcome statistics come from regression models, this seems the most useful approach to me. I know that you wrote "most of the time" :-) $\endgroup$ – Bernd Weiss Jun 12 '12 at 7:14
6
$\begingroup$

There already exist approaches that aim at synthesizing individual and aggregate person data. The Sutton et al. (2008) paper applies a Bayesian approach which (IMHO) has some similarities to your idea.

  • Riley, R. D., Lambert, P. C., Staessen, J. A., Wang, J., Gueyffier, F., Thijs, L., & Boutitie, F. (2007). Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Statistics in Medicine, 27(11), 1870–1893. doi:10.1002/sim.3165 PDF

  • Riley, R. D., & Steyerberg, E. W. (2010). Meta‐analysis of a binary outcome using individual participant data and aggregate data. Research Synthesis Methods, 1(1), 2–19. doi:10.1002/jrsm.4

  • Sutton, A. J., Kendrick, D., & Coupland, C. A. C. (2008). Meta-analysis of individual- and aggregate-level data. Statistics in Medicine, 27(5), 651–669.

$\endgroup$
10
$\begingroup$

I thank @Bernd for pointing me in the right direction. Here are some notes on the references he mentioned in his answer, as well as some of the references mentioned in these articles.

Sutton et al (2008)

Sutton et al use within a health context the terms individual patient data versus aggregate data.

They note that analysis of individual patient data is often considered to be the gold standard for meta-analysis, citing Stewart and Clark (1995). It is particularly useful for assessing data quality and performing analyses on values not reported in existing reports (e.g., particular subgroup analyses). Naturally, they note problems, such as the impossibility in some cases of obtaining all individual patient data and the additional costs in processing such data. They also observe that for simple models where the summary statistics are available results will often be similar or the same.

They also observe the infrequency of individual patient meta-analysis citing a review by Simmonds et al (2005). They also mention the the review article of meta-analysis combining individual patient data with aggregate data by Riley RD, Simmonds, et al (2008)

Riley Lambert Abo-Zaid (2010)

In this article Riley et al describe more about meta-analysis of individual participant data. They outline advantages of meta-analysis of individual participant data (e.g., consistent data processing, modelling of missing data, verification of original reported results, more analysis options, etc.)

Stewart & Tierney (2002)

Stewart and Tierney review the pros and cons of individual patient data meta-analysis focusing particularly on practical issues.

Riley Lambert et al (2007)

They describe methods of combining individual patient data with aggregate data in terms of one-step and two-step approaches.

Cooper & Patall (2009)

Cooper and Patall wrote an article as part of a special issue on meta-analysis of individual-level data in Psychological Methods (see Shrout, 2009 for a summary). Cooper and Patall describe research synthesis as one in a second stage of transition:

The first transition is from the narrative research review—in which opaque rules of cognitive algebra are used to synthesize the results of studies—to meta-analysis of [aggregated data]. The second stage involves the transition from meta-analysis of [aggregated data] to the accumulation of [individual participant-level data].

to be continued...

References

  • Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological methods, 14(2), 165–176. doi:10.1037/a0015565
  • Riley, R. D., Lambert, P. C., Staessen, J. A., Wang, J., Gueyffier, F., Thijs, L., & Boutitie, F. (2007). Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Statistics in Medicine, 27(11), 1870–1893. doi:10.1002/sim.3165 [PDF] (http://www.staessen.net/publications/2006-2010/08-21-P.pdf)
  • Riley, R. D., Lambert, P. C., & Abo-Zaid, G. (2010). Meta-analysis of individual participant data: rationale, conduct, and reporting, BMJ, 340, 221.
  • Riley RD, Simmonds MC, Look MP. (2007) Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. Journal of Clinical Epidemiology , in press and early view.
  • Riley, R. D., & Steyerberg, E. W. (2010). Meta‐analysis of a binary outcome using individual participant data and aggregate data. Research Synthesis Methods, 1(1), 2–19. doi:10.1002/jrsm.4
  • Shrout, P.E. (2009). Short and long views of integrative data analysis: Comments on contributions to the special issue.. Psychological methods, 14, 177.
  • Simmonds MC, Higgins JPT, Stewart LA, Tierney JF, Clarke MJ, Thompson SG. (2005). Meta-analysis of individual patient data from randomized trials: a review of methods used in practice. Clinical Trials ; 2:209–217.
  • Stewart LA, Clarke MJ. Practical methodology of meta-analyses (overviews) using updated individual patient data. Cochrane Working Group. Statistics in Medicine 1995; 14:2057–2079.
  • Stewart LA, Tierney JF. To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Eval Health Prof 2002;25:76-97.
  • Sutton, A. J., Kendrick, D., & Coupland, C. A. C. (2008). Meta-analysis of individual- and aggregate-level data. Statistics in Medicine, 27(5), 651–669.
$\endgroup$
  • $\begingroup$ Great answer, Jeromy! Actually, this should become the best answer... Here are a few more papers that are missing in your list. $\endgroup$ – Bernd Weiss Jun 12 '12 at 8:00
  • $\begingroup$ Thanks. I'm just taking some notes as I read through the references. Those additional references are particularly useful thanks. $\endgroup$ – Jeromy Anglim Jun 12 '12 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.