23
$\begingroup$

As science must be reproducible, by definition, there is increasing recognition that data and code are an essential component of the reproduciblity, as discussed by the Yale Roundtable for data and code sharing.

In reviewing a manuscript for a journal that does not require data and code sharing, can I request that the data and code be made available

  1. to me at the time of review
  2. publicly at time of publication (the journal supports supplements)

also, how might I phrase such a request?

update: although I am interested in the general case, this particular case consists of a meta-analysis with all previously published data, and the code is simple linear models in SAS

side note the ability to make cross-study inference (as is the goal of meta-analysis) would be greatly enhanced if more studies provided raw data

update 2:

I requested the data and code from the editor for purposes of review, the editor considered the request reasonable, and I have received the requested material (sufficient but with cryptic variable names, no metadata, and few inline comments) within a day.

Improve this question
$\endgroup$
12
  • 3
    $\begingroup$ Personally, I would be ok with code in pretty much any circumstance. Data is another issue. Unless it is already publicly available, I likely won't have permission to share it with anyone. I suspect this makes a data request unreasonable in many circumstances. $\endgroup$
    – Andy W
    Aug 17, 2011 at 17:00
  • 4
    $\begingroup$ This sounds like something you should discuss with your associate editor or editor. If they're doing their job, they should be able to provide you with guidance and would probably serve as the point of contact for the authors for any such request. $\endgroup$
    – cardinal
    Aug 17, 2011 at 18:26
  • 1
    $\begingroup$ It seems to me that many of the examples and ensuing discussion are based on anecdotal situations that can vary tremendously according to discipline and what data we are talking about. Sure some situations seem perfectly reasonable to disseminate data, but that doesn't necessarily characterize all situations. This question is turning into a rant very quickly! $\endgroup$
    – Andy W
    Aug 17, 2011 at 20:37
  • 1
    $\begingroup$ @Andy any suggestions about how to steer away from a rant? e.g. distinctions that should be made in the OP or edits to the answers (it is CW after all). $\endgroup$
    – Abe
    Aug 17, 2011 at 21:01
  • 1
    $\begingroup$ I think this form has a selection bias :-). Most of us are here because most of us support OPs idea in some way. $\endgroup$
    – suncoolsu
    Aug 19, 2011 at 3:43

4 Answers 4

Reset to default
7
$\begingroup$

As far as getting data as a reviewer goes, you're entitled to it if you need it to complete your review properly. More reviewers should be asking for data and assessing it. Lots of journals have policies that they may require the data and analysis code for review purposes.

Availability at the time of publication isn't clear to me. It seems that you're saying that you want to force the issue that the data be made publicly available as a condition of publication. That's a bad idea if it's not journal policy already. You're making publication an unfair moving target. They submitted expecting that not to be a requirement and you, nor the editor, ought to be changing the game.

Unbeknownst to many researchers publicly funded researchers, they are required to make their data publicly available. For example, most NIH grants have clauses where the researcher must be forthcoming with their data. Most government granting agencies have data sharing clauses that force the researcher to share what they find (perhaps force is a bit strong given that it's very hard to lose a grant over that... perhaps lose renewal though). The public paid for the data, therefore the public is entitled to it---in the case of human research, entitled to it anonymized.

Some of the most expensive and sensitive data to collect, human FMRI data, is also some of the most commonly made publicly available. Not just PLoS, but major journals of the field require the submission of the data and maintain a publicly available data bank. I think this says a lot to people who object for reasons of cost (it's very expensive), and privacy (it's human data from small studies and sometimes unique clinical populations that could be very sensitive). Those are reasons that make that data more valuable to the public. Researchers who withhold such data are doing a disservice to the people who bought it (everyone), and need a lesson in what their responsibilities are outside of their little lab and publication competition.

If the research was privately funded, genuinely privately funded, then best of luck.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I agree with most of what's in this - there are already requirements to disclose data - usually to other qualified researchers - in NIH funded grants and the like. But I think extending this to an ad hoc requirement for publication is vastly overstepping the role of a reviewer. $\endgroup$
    – Fomite
    Aug 17, 2011 at 20:46
  • 1
    $\begingroup$ (+1) Thoughful, well-articulated answer. One thing to keep in mind is that this site has a fairly international audience. I would hope more researchers would take a collaborative view toward their research and data even if there is no NIH or other organization twisting their arm. $\endgroup$
    – cardinal
    Aug 17, 2011 at 20:49
  • $\begingroup$ EpiGrad, in case it's not clear from the answer, I felt making an ad hoc requirement for publication not only overstepped the bounds of the reviewer, but the editor as well. $\endgroup$
    – John
    Aug 17, 2011 at 21:01
  • $\begingroup$ Cardinal, thanks for the compliment. Thank for the reminder as well. I do try to keep in mind that the site is international. As big as NIH is, it isn't in my country. ;) $\endgroup$
    – John
    Aug 17, 2011 at 22:31
  • $\begingroup$ @John. Yeah - I just forgot to put +1 while agreeing with you :) $\endgroup$
    – Fomite
    Aug 18, 2011 at 0:12
6
$\begingroup$

Addressing the two situations seperately:

As a reviewer: Yes, I think you'd have grounds to ask to see the data or the code. But if I were you, I'd prepare to see things like pared down code, or a subsample of the data. People implement future research not being reported in this paper in their code all the time, and you've no entitlement to said code. Since I do mostly biomedical research, I'd also be prepared to have to deal with some fairly restrictive data use agreements.

In the journal itself: No. If a researcher wants to reproduce my results, they can approach me themselves to ask for code - that's why we have corresponding authors. For data, absolutely not, under no circumstances. My data is governed by IRB and confidentiality agreements - it's not just going to be made public. If I want a public-ish data set, I might simulate a dataset with similar properties (i.e. the "Faux-Mesa" network data available in one of the network packages for R), but as a reviewer, you've got no call to force that. If its a journal-wide requirement, then the authors knew their data/code would be public when submitting it, but if its not then no. Your role is to evaluate the quality of the paper itself (hence my being alright with it for the purposes of the review), not use your ability to contribute to the acceptance/rejection of the paper to push what is essentially a philosophical/political point outside the scope of the journal.

At best, I'd put a "I would strongly urge the authors to make their code and data available, where possible" in your comments, but I wouldn't phrase it any stronger than that, and I wouldn't put it in the formal list of "Things I think need fixing before this sees the light of day".

Improve this answer
$\endgroup$
10
  • $\begingroup$ Just noticed your edits: In this case, as far as the paper is concerned, by answer is actually a stronger 'no' - assuming the paper is has citations. If the point of the request is 'reproducible research', and the data is publically findable, then there is no reason why a researcher seeking to confirm findings could not do this themselves. Furthermore, if the point is actually evaluate the science, rather than merely demonstrate that you too can click "Run" and get the same results, going through the searching and abstraction process in the literaute yourself is part of verifying a result. $\endgroup$
    – Fomite
    Aug 17, 2011 at 18:23
  • $\begingroup$ for research to be reproducible, the version of the code and data used in the analysis should be made available, and code not used in the analysis would not be expected (or necessarily relevant). $\endgroup$
    – David LeBauer
    Aug 17, 2011 at 18:42
  • $\begingroup$ @EpiGrad: On some level, I don't quite agree with this strong no. Keith Baggerly has been evangelizing on this subject lately and makes some interesting observations. See, for example, K. A. Baggerly and K. R. Coombes, Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology, Ann. Appl. Stat., vol 3, no. 4, pp. 1309-1334. There is also an arxiv version. $\endgroup$
    – cardinal
    Aug 17, 2011 at 19:02
  • $\begingroup$ I know of very few grant funded researchers who have IRB and confidentiality issues that genuinely restrict the publication of properly anonymized data. If you're in that situation it's a pretty rare case. $\endgroup$
    – John
    Aug 17, 2011 at 19:28
  • $\begingroup$ @EpiGrad I also question the strong no. The methods used to get from raw data to derived metric in a meta-analysis often require a large suite of assumptions related to differences in methodology and interpretation that occur on the level of an individual study. In this case, there are over 200 studies represented, so the time required to reconstruct the data set would be prohibitive - and would ultimately inhibit the ultimate goal of progressing science. $\endgroup$
    – David LeBauer
    Aug 17, 2011 at 19:59
5
$\begingroup$

As John says availability of data to reviewers should be a no-brainer; careful review should include replicating the analysis and as such necessitates access to the data.

With regards to public availability of the data following publication, I'd say that battle should be fought with the journal generally rather than with regards to a specific submission.

On a more general note, funding agencies and IRBs are becoming increasingly aware that data sharing is both scientifically and ethically necessary component of research. By increasing the availability for re-analysis that could yield new results of correct erroneous reports, data sharing increases the potential benefits to research, thereby modifying the cost/benefit tradeoff to the advantage of the participants of the research. Certainly it is necessary to inform participants of the possibility that their data will be shared, and it is also necessary to set up safeguards to prevent increased risk of identification to participants, but these can be achieved in most circumstances. In my own research, I assure participants (and my IRB) that (1) data will be stored in a strong encrypted format (updated as decryption technology advances), (2) data will be shared with qualified researchers upon request, but only if they agree (3) to similarly store the data in a strong encrypted format (updated as decryption technology advances), (4) refrain from sharing the data (instead referring requests to me), and (5) refrain from connecting the data with data from any other sources unless (6) the data connection is explicitly permitted by an IRB, who would determine whether the connection would unacceptably (relative to the potential benefits of the project) increase the risk of identifiability.

Improve this answer
$\endgroup$
1
$\begingroup$

I don't have any experience with this, but it seems to me that you might be able to insist on #1 as a part of your own due diligence in reviewing their results. I don't see how you can insist on #2, though.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.