How to handle studies that don’t provide an estimate of effect size in a meta-analysis?

Question

Say we want to survey the literature to find out the effect of Treatment A on Value B.

Some studies provide an effect size estimate with confidence intervals and t-test results (e.g., “Treatment A had a significant effect on Value B (mean effect size = 3.5, 95% C.I. = [3, 4], p-value < 0.01”).

Other studies just provide the p-value from the t-test results (e.g., “Treatment A did not have a significant effect on Value B (p-value = 0.99)”).

How can the results from this second group of studies be included in a meta-analysis (if they can be included at all)?

Since there is no effect size given, I can’t really think of a way to include them. At the same time, to exclude them seems problematic (especially because it seems to me that the studies that find no significant effect are most likely to report the results in this way).

Right now my working solution is to keep track of this second group of studies, make a summary table regarding their conclusions, and present that along with the meta-analysis of the studies that do present effect size.

What are better solutions?

EDIT: Some clarification in response to a comment. Here are some examples of situations where effect size has not been presented.

Sometimes there's no information presented other than saying "there was no difference".

Sometimes there are p-values.

Sometimes there are tables full of p-values.

But no effect sizes or raw data with which to calculate effect sizes. What's a person to do?

Answer 1

There are two main strategies which have been suggested in the literature.

One possibility is to use some form of imputation. There are two R packages available from CRAN which take different approaches. SAMURAI assumes you have some information from the article about whether the authors saw their results as very positive, positive, and so on but did not present the effect size. metansue does meta-analysis of Non Significant Unreported Effects which as the name suggests is the situation where you know how many there were but only that they were not significant at some level. The authors of the package have published on their methods here

A second possibility is to meta-analyse the $p$ values using one of the many methods for that (Fisher, Lancaster, Stouffer, Tippett, ...). These have been compared by Loughin here. In this case you do nt get an effect size, obviously, but you do get an overall $p$-value.

There are links to the packages mentioned and also to a number which meta-analyse $p$-values in the CRAN Task View on MetaAnalysis