3
$\begingroup$

One common practice which increases the odds of obtaining spurious results is to keep collecting observations after a preliminary analyses are performed[1]. This occurs when the cutoff point for collecting cases is set as the time when significance is reached.

Noting that, it seems that meta analyses are a valid way of combining a variety of studies in a way that minimises overall error in the parameter estimate. But I guess that it's also a result of not having access to the original data.

So my question is, what if you do have access to the original data from previous experiments? Is it better to do a meta analysis of the results or to combine the data from the previous studies and run the analyses again, apparently called individual participant data (IPD),?

My guess is that the latter is ill-advised although some people do it. For one, it appears to be suspiciously close to the undefined cutoff point discussed in Simmons et al. It also seems to go against other advice I have received on this site regarding dipping into the same data set multiple times.

Note that in my particular case, I may have access to summary data of each participant in a study I replicated, not all data points collected for every participant. Essentially I may have access to means, totals, etc for each participant in a previous study.


[1]: see, for example, Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. doi:10.1177/0956797611417632

$\endgroup$
1
$\begingroup$

As the linked post How to conduct a meta-analysis using raw data? mentions this is widely done. If there is a concern about doing this sort of sequential analysis one might look at some of the papers which have tried to put this into a sequential framework. For instance Wetterslev and colleagues in J Clin Epid here or a follow-up here and another one in IJE here. Some of these may be behind a paywall but they seem to be widely copied in multiple internet sites.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.