32
$\begingroup$

Several methodological papers (e.g. Egger et al 1997a, 1997b) discuss publication bias as revealed by meta-analyses, using funnel plots such as the one below. Funnel plot of beta blockers in myocardial infarction

The 1997b paper goes on to say that "if publication bias is present, it is expected that, of published studies, the largest ones will report the smallest effects." But why is that? It seems to me that all this would prove is what we already know: small effects are only detectable with large sample sizes; while saying nothing about the studies that remained unpublished.

Also, the cited work claims that asymmetry that is visually assessed in a funnel plot "indicates that there was selective non-publication of smaller trials with less sizeable benefit." But, again, I don't understand how any features of studies that were published can possibly tell us anything (allow us to make inferences) about works that were not published!

References
Egger, M., Smith, G. D., & Phillips, A. N. (1997). Meta-analysis: principles and procedures. BMJ, 315(7121), 1533-1537.

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629-634.

$\endgroup$
  • $\begingroup$ I do not think you have this the right way round. Perhaps the answer to this Q&A might help stats.stackexchange.com/questions/214017/… $\endgroup$ – mdewey Feb 19 at 13:50
  • 7
    $\begingroup$ For a small study to get published at all it will have to show a large effect no matter what the true effect size is. $\endgroup$ – einar Feb 19 at 13:52
23
$\begingroup$

The answers here are good, +1 to all. I just wanted to show how this effect might look in funnel plot terms in an extreme case. Below I simulate a small effect as $N(.01, .1)$ and draw samples between 2 and 2000 observations in size.

The grey points in the plot would not be published under a strict $p < .05$ regime. The grey line is a regression of effect size on sample size including the "bad p-value" studies, while the red one excludes these. The black line shows the true effect.

As you can see, under publication bias there is a strong tendency for small studies to overestimate effect sizes and for the larger ones to report effect sizes closer to the truth.

set.seed(20-02-19)

n_studies <- 1000
sample_size <- sample(2:2000, n_studies, replace=T)

studies <- plyr::aaply(sample_size, 1, function(size) {
  dat <- rnorm(size, mean = .01, sd = .1)
  c(effect_size=mean(dat), p_value=t.test(dat)$p.value)
})

studies <- cbind(studies, sample_size=log(sample_size))

include <- studies[, "p_value"] < .05

plot(studies[, "sample_size"], studies[, "effect_size"], 
     xlab = "log(sample size)", ylab="effect size",
     col=ifelse(include, "black", "grey"), pch=20)
lines(lowess(x = studies[, "sample_size"], studies[, "effect_size"]), col="grey", lwd=2)
lines(lowess(x = studies[include, "sample_size"], studies[include, "effect_size"]), col="red", lwd=2)
abline(h=.01)

Created on 2019-02-20 by the reprex package (v0.2.1)

$\endgroup$
  • 1
    $\begingroup$ Excellent point, really helps understanding this intuitively, thanks! $\endgroup$ – z8080 Feb 20 at 13:31
  • 2
    $\begingroup$ +1 This graphic is worth a thousand words and summarizes the problem well. This type of bias can even be found when the true effect size is 0. $\endgroup$ – Underminer Feb 20 at 19:18
29
$\begingroup$

First, we need think about what "publication bias" is, and how it will affect what actually makes it into the literature.

A fairly simple model for publication bias is that we collect some data and if $p < 0.05$, we publish. Otherwise, we don't. So how does this affect what we see in the literature? Well, for one, it guarantees that $|\hat \theta |/ SE(\hat \theta) >1.96$ (assuming a Wald statistic is used). Now, one point being made is that if $n$ is really small, then $SE(\hat \theta)$ is relatively large and a large $|\hat \theta|$ is required for publication.

Now suppose that in reality, $\theta$ is relatively small. Suppose we run 200 experiments, 100 with really small sample sizes and 100 with really large sample sizes. Note that of 100 really small sample size experiments, the only ones that will get published by our simple publication bias model is those with large values of $|\hat \theta|$ just due to random error. However, in our 100 experiments with large sample sizes, much smaller values of $\hat \theta$ will be published. So if the larger experiments systematically show smaller effect than the smaller experiments, this suggests that perhaps $|\theta|$ is actually significantly smaller than what we typically see from the smaller experiments that actually make it into publication.

Technical note: it's true that either having a large $|\hat \theta|$ and/or small $SE(\hat \theta)$ will lead to $p < 0.05$. However, since effect sizes are typically thought of as relative to standard deviation of error term, these two conditions are essentially equivalent.

$\endgroup$
  • $\begingroup$ "Now, one point being made is that if $n$ is really small, then $SE(\theta)$ is relatively large and a large $|\theta|$ is required for publication." This is not, technically speaking, necessarily true: $SE(\theta) = \frac{SD(\theta)}{\sqrt{n}}$: if $SE(\theta)$ is very small, then a small $SE$ may result even for a small sample size, right? EDIT: Oh wait! Just read your closing sentence. :) +1 $\endgroup$ – Alexis Feb 19 at 22:13
19
$\begingroup$

Read this statement a different way:

If there is no publication bias, effect size should be independent of study size.

That is, if you are studying one phenomenon, the effect size is a property of the phenomenon, not the sample/study.

Estimates of effect size could (and will) vary across studies, but if there is a systematic decreasing effect size with increasing study size, that suggests there is bias. The whole point is that this relationship suggests that there are additional small studies showing low effect size that have not been published, and if they were published and therefore could be included in a meta analysis, the overall impression would be that the effect size is smaller than what is estimated from the published subset of studies.

The variance of the effect size estimates across studies will depend on sample size, but you should see an equal number of under and over estimates at low sample sizes if there were no bias.

$\endgroup$
  • 1
    $\begingroup$ But is it really correct to say that "If there is no publication bias, effect size should be independent of study size"? This is true of course when you refer to the true underlying effect, but I think they are referring to the estimated effect. An effect size that is dependent of study size (suggesting bias) amounts to a linear relationship in that scatter plot (high correlation). This is something I have personally not seen in any funnel plots, even though of course many of those funnel plots did imply that a bias existed. $\endgroup$ – z8080 Feb 19 at 18:38
  • 2
    $\begingroup$ @z8080 You're right, only if the mean and standard deviation estimates are unbiased will the estimated effect size be completely independent of study size if there is no publication bias. Since the sample standard deviation is biased, there will be some bias in the effect size estimates, but that bias is small compared to the level of bias across studies that Egger et al are referring to. In my answer I'm treating it as negligible, assuming the sample size is large enough that the SD estimate is nearly unbiased, and therefore considering it to be independent of study size. $\endgroup$ – Bryan Krause Feb 19 at 19:44
  • 2
    $\begingroup$ @z8080 The variance of the effect size estimates will depend on sample size, but you should see an equal number of under and over estimates at low sample sizes. $\endgroup$ – Bryan Krause Feb 19 at 19:46
  • 2
    $\begingroup$ "Estimates of effect size could (and will) vary across studies, but if there is a systematic relationship between effect size and study size" That phrasing is a bit unclear about the difference between dependence and effect size. The distribution of effect size will the different for difference sample size, and thus will not be independent of sample size, regardless of whether there's bias. Bias is a systematic direction of the dependence. $\endgroup$ – Acccumulation Feb 19 at 23:11
  • $\begingroup$ @Acccumulation Does my edit fix the lack of clarity you saw? $\endgroup$ – Bryan Krause Feb 19 at 23:23

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.