3
$\begingroup$

Say we want to survey the literature to find out the effect of Treatment A on Value B.

Some studies provide an effect size estimate with confidence intervals and t-test results (e.g., “Treatment A had a significant effect on Value B (mean effect size = 3.5, 95% C.I. = [3, 4], p-value < 0.01”).

Other studies just provide the p-value from the t-test results (e.g., “Treatment A did not have a significant effect on Value B (p-value = 0.99)”).

How can the results from this second group of studies be included in a meta-analysis (if they can be included at all)?

Since there is no effect size given, I can’t really think of a way to include them. At the same time, to exclude them seems problematic (especially because it seems to me that the studies that find no significant effect are most likely to report the results in this way).

Right now my working solution is to keep track of this second group of studies, make a summary table regarding their conclusions, and present that along with the meta-analysis of the studies that do present effect size.

What are better solutions?

EDIT: Some clarification in response to a comment. Here are some examples of situations where effect size has not been presented.

Sometimes there's no information presented other than saying "there was no difference". Weed density and community
Corn yield Soil water

Sometimes there are p-values.

Soybean yield

Sometimes there are tables full of p-values.

Surface soil properties P-value table

But no effect sizes or raw data with which to calculate effect sizes. What's a person to do?

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Are you saying they have provided no information other than p-values? No raw data, no parameter estimates? $\endgroup$ – mkt - Reinstate Monica Jul 29 '19 at 20:26
  • $\begingroup$ @mkt, yes, often there is no information provided other than p-values, and sometimes not even that. I've edited my question to show specific examples. $\endgroup$ – Angela Jul 30 '19 at 1:22
  • 1
    $\begingroup$ That is unfortunate and it certainly weakens your ability to do an unbiased meta-analysis; this seems like a weaker form of the file-drawer effect. I imagine you would see bias in a funnel plot of the significant results. I can't think of a great solution, but perhaps some form of mixture modelling approach might be useful. $\endgroup$ – mkt - Reinstate Monica Jul 30 '19 at 7:51
  • 1
    $\begingroup$ There is no silver bullet solution here. First start by contacting the authors for the unreported data. Even if they provide it, it hasn't been peer-reviewed and that can introduce bias. If not available, or no response to the request, you can't make any valid assumptions on what the values were. Therefore, report this narratively or exclude from your study due to no analyzable data. In all cases, be transparent on how you dealt with these kinds of studies as it may be a cause of publication bias as mentioned by @mkt. $\endgroup$ – abousetta Jul 30 '19 at 11:50
4
$\begingroup$

There are two main strategies which have been suggested in the literature.

One possibility is to use some form of imputation. There are two R packages available from CRAN which take different approaches. SAMURAI assumes you have some information from the article about whether the authors saw their results as very positive, positive, and so on but did not present the effect size. metansue does meta-analysis of Non Significant Unreported Effects which as the name suggests is the situation where you know how many there were but only that they were not significant at some level. The authors of the package have published on their methods here

A second possibility is to meta-analyse the $p$ values using one of the many methods for that (Fisher, Lancaster, Stouffer, Tippett, ...). These have been compared by Loughin here. In this case you do nt get an effect size, obviously, but you do get an overall $p$-value.

There are links to the packages mentioned and also to a number which meta-analyse $p$-values in the CRAN Task View on MetaAnalysis

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.