# How to handle studies that don’t provide an estimate of effect size in a meta-analysis?

Say we want to survey the literature to find out the effect of Treatment A on Value B.

Some studies provide an effect size estimate with confidence intervals and t-test results (e.g., “Treatment A had a significant effect on Value B (mean effect size = 3.5, 95% C.I. = [3, 4], p-value < 0.01”).

Other studies just provide the p-value from the t-test results (e.g., “Treatment A did not have a significant effect on Value B (p-value = 0.99)”).

How can the results from this second group of studies be included in a meta-analysis (if they can be included at all)?

Since there is no effect size given, I can’t really think of a way to include them. At the same time, to exclude them seems problematic (especially because it seems to me that the studies that find no significant effect are most likely to report the results in this way).

Right now my working solution is to keep track of this second group of studies, make a summary table regarding their conclusions, and present that along with the meta-analysis of the studies that do present effect size.

What are better solutions?

EDIT: Some clarification in response to a comment. Here are some examples of situations where effect size has not been presented.

Sometimes there's no information presented other than saying "there was no difference".

Sometimes there are p-values.

Sometimes there are tables full of p-values.

But no effect sizes or raw data with which to calculate effect sizes. What's a person to do?

• Are you saying they have provided no information other than p-values? No raw data, no parameter estimates? – mkt - Reinstate Monica Jul 29 at 20:26
• @mkt, yes, often there is no information provided other than p-values, and sometimes not even that. I've edited my question to show specific examples. – Angela Jul 30 at 1:22
• That is unfortunate and it certainly weakens your ability to do an unbiased meta-analysis; this seems like a weaker form of the file-drawer effect. I imagine you would see bias in a funnel plot of the significant results. I can't think of a great solution, but perhaps some form of mixture modelling approach might be useful. – mkt - Reinstate Monica Jul 30 at 7:51
• There is no silver bullet solution here. First start by contacting the authors for the unreported data. Even if they provide it, it hasn't been peer-reviewed and that can introduce bias. If not available, or no response to the request, you can't make any valid assumptions on what the values were. Therefore, report this narratively or exclude from your study due to no analyzable data. In all cases, be transparent on how you dealt with these kinds of studies as it may be a cause of publication bias as mentioned by @mkt. – abousetta Jul 30 at 11:50

A second possibility is to meta-analyse the $$p$$ values using one of the many methods for that (Fisher, Lancaster, Stouffer, Tippett, ...). These have been compared by Loughin here. In this case you do nt get an effect size, obviously, but you do get an overall $$p$$-value.
There are links to the packages mentioned and also to a number which meta-analyse $$p$$-values in the CRAN Task View on MetaAnalysis