# Qualitative assessment of (negative) publication bias

I have been doing some literature analysis in a certain surgical implants. In the early 2000´s Implant 1 was introduced. This implant proved to be successful and naturally several medical industries made their own versions. The copycats were launched some years later. Unfortunately these copycats were disasters, they failed (=required removal) in large numbers and were subsequently withdrawn from the market. Implant 1, the original, still works fine.

Many countries have their own national implant registries which record and track the survival of these implants. With regard to these certain implants, these registries were credited "catching" these failing implants and minimizing the damage. In my profession during the last years I noticed early that there were practically no studies reporting the results of these copycats, only the results of the original were reported.

As so, I collected data from two large national registries which have recorded the use of these implants. Also I looked all the studies which have reported the use of these implants. Here is my data:

   Year Imp1Reg Imp1Lit Imp1RegRel Imp1LitRel Imp2Reg Imp2Lit Imp2RegRel Imp2LitRel Imp3Reg Imp3Lit Imp3RegRel Imp3LitRel
1  2003    3238       0  0.1009006 0.00000000      53       0 0.01171789 0.00000000       0       0 0.00000000  0.0000000
2  2004    7965     446  0.2482004 0.02744109     474       0 0.10479770 0.00000000     263       0 0.03596335  0.0000000
3  2005   12741    1304  0.3970272 0.08023134    1502       0 0.33208048 0.00000000    2341       0 0.32011486  0.0000000
4  2006   17013    1358  0.5301486 0.08355380    2579       0 0.57019677 0.00000000    4495       0 0.61465883  0.0000000
5  2007   20934    2118  0.6523324 0.13031440    3520       0 0.77824453 0.00000000    6519       0 0.89142623  0.0000000
6  2008   24152    3368  0.7526098 0.20722328    4178     115 0.92372319 0.07376523    7273       0 0.99453029  0.0000000
7  2009   26767    5197  0.8340968 0.31975635    4482     559 0.99093522 0.35856318    7304       0 0.99876931  0.0000000
8  2010   28677    7947  0.8936150 0.48895588    4523     773 1.00000000 0.49583066    7312     301 0.99986326  0.3058943
9  2011   30154   14315  0.9396404 0.88076047    4523    1262 1.00000000 0.80949326    7313     401 1.00000000  0.4075203
10 2012   31174   15528  0.9714250 0.95539285    4523    1315 1.00000000 0.84348942    7313     862 1.00000000  0.8760163
11 2013   32091   16253  1.0000000 1.00000000    4523    1559 1.00000000 1.00000000    7313     984 1.00000000  1.0000000


Each implant has the cumulative number of patients reported in the registries (Reg) and in the literature (Lit). From each variable I have derived a relative number which indicates the cumulative proportion of patients each year reported in the literature or entered in the registries. For example for Implant 1, the cumulative proportion in year 2007 in the literature is 2118/16253=0.1303144 and for registry 20934/32091=0.6523324.

It is evident that I cant have the numbers of the implants used in the whole world, but with these two registries from two countries with a population close to 100 million I am fairly certain I gave a good estimate for the trends in the use around the world. As so it is not reasonable to compare the absolute numbers.

As with any new medical innovation we should gather data, analyze it and then justify the use of this innovation.

In order to investigate to fulfillment of this practice I plotted the proportion of patients entered yearly in the registries against the proportion of patients reported yearly in the literature.

Implant 1:

Circles indicate the literature and dots indicate the registries.

My interpretation is that soon after the launch of the Implant 1, increasing number of studies is published in the literature (=evidence). Of course these phenomenons cannot happen simultaneously.

For Implant 2 (copycat) the graph is different:

First studies are published only when the use of the implant has practically stopped. Not very wise, "use in great number, investigate later".

For Implant 3 the history is far worse:

After the use of the implant has been stopped the first studies are published.

So, my question is that, is there a way to quantitatively or in any other way to assess this negative bias which is clearly observed in the literature?

I would plot the proportions against each other. Implant 1:

Implant 2:

Implant 3:

For Implant 1 we can see that the line is close to diagonal (Maybe diagoal line would be the best?). This would be optimal, the interpretation would be that, as we use the implant, at the same time studies are also published investigating the performance of the implant (any study is good, regardless of outcome).

But for Implant 2 the line is very convex due to the fact that, while the implant have been used greatly, there is only very limited number of studies. And for Implant 3 the line is "collapsed". This was due to fact that we first used the implant in very large numbers. Then it was recalled and investigation was performed only after the disaster was imminent.

Any comments on my own method? And more importantly, has someone described this method this earlier? Thanks!

• Very interesting research – Aksakal Jul 28 '15 at 18:25

Strictly speaking, the way it is formulated here, this question it is less of a publication bias (which more commonly refers to the decreased likelihood of people trying to publish, and journals accepting, findings that are either at odds with the current paradigm or that are negative/inconclusive) than analyzing time-lag of publications for the different types of treatments/implants.

The figures you made are great for illustrating the trends and perhaps hypothesis test is not necessary in this case. But if you want to analyze these, replication would be critical: e.g. if you can deconstruct the data to have replication within each year, you could examine temporal trends more specifically using repeated measures analysis. Without that, you could focus only on the years of purported lag (eg omitting 2012-13, when they all converge), express each datapoint as a rate (the ratio of relative increases in both proportions) of 'literature acceptance' or the difference between the two ratios and analyze it using e.g. a permutation t-test. Admittedly, a suboptimal approach because of the obvious subjective a posteriori 'trimming'.

Aside from that, skewed effect sizes can be one of the indications of publication bias, and increasing time lag between reported year of the study and publication date could be indicative of negative results sitting in the drawer for too long, waiting for a flood of problems to arise, or potentially even being suppressed by the funding source.