8
$\begingroup$

Im trying to understand how people use R packages and was wondering if there are documented cases where R packages have produced different answers.

Clarification: The motivation behind this question comes from an effort I'm involved in where the goal to understand the importance of provenance in the analytical methods and how it facilitates reproducible research. While R is big in the science community at present, and R packages are versioned in CRAN, without detailed information [especially version numbers], someone trying to reproduce a body of work in the future might come to a different conclusion that the original work (even with the original data).

Example: Paper by John Doe says "we used R 2.3.1 and package glmulti to fit our models". 10 years from now, someone might use a new version of glmulti (no one knows what version was used in the original) which might produce a much different conclusion. My question: Are there examples of such a thing happening already? Version 2 or R package produces a much different result that version 1.

$\endgroup$
2
  • 1
    $\begingroup$ The question is a tad vague. Can you focus it a bit more? $\endgroup$ Oct 17, 2011 at 22:49
  • $\begingroup$ Yes, I clarified the question. $\endgroup$
    – Maiasaura
    Oct 18, 2011 at 14:22

4 Answers 4

6
$\begingroup$

I've had issues with package glmnet over versions. If I remember well, it was going from version 1.5 to 1.6, but I may be off a little.

The package creators/maintainers changed the order of the classes of their objects (so it became c("lognet", "glmnet") instead of c("glmnet", "lognet") - or it may have been the other way around). Of course, they also changed all their S3 functions to properly handle this (e.g. predict.lognet).

The problem with this was: an object you had created with an old version of glmnet was not compatible with the new functions (since dispatching worked the other way around). Most people wouldn't be in that spot (who saves a glmnet object for later use?), but I was.

Mind you: this is a very powerful package, developed by extremely intelligent people, so it could happen to anyone :-)

$\endgroup$
1
  • 1
    $\begingroup$ To answer who saves objects: I do. :) $\endgroup$
    – Iterator
    Nov 10, 2011 at 19:13
8
$\begingroup$

This will vary package to package, but the general answer is yes. Outputs can vary, and even basic usage too (input/ouput args). This is why, when I do an important analysis, I always like to document what versions were used with version() and sessionInfo(). Even if things change, old versions are retained on CRAN, so you can get the old versions if you need them.

$\endgroup$
2
  • 1
    $\begingroup$ +1 Great advice. I have been in an R tutorial class before where the instructor's code was written using an older version of R than the campus lab computers had and it took 30 minutes of debugging to figure out what changed in the point releases and how the course code had to be modified. While some software is perhaps too obedient to backwards compatibility it has never struck me that this would be a criticism of R. $\endgroup$ Oct 18, 2011 at 1:52
  • 1
    $\begingroup$ @JohnColby is correct - it is very important to be able to reproduce your setup, including package version numbers, and including things that are often overlooked, such as the arguments passed to functions (I try to always use named arguments), the random number seed, the various external dependencies, and more. As for whether any one package has affected results, the answer is yes, and broader than you might expect: even basic I/O packages can affect results if you lose data. :) You might lose data or the data may be modified in some way if the default file loading behavior has changed. $\endgroup$
    – Iterator
    Nov 10, 2011 at 19:16
3
$\begingroup$

Just a quick point:

  • The R package ecosystem is very big, and it's really up to the individual author, whether they intend to maintain backwards compatibility.
  • I personally haven't had any issues with base R packages changing in ways that led to issues of backwards compatibility. In general, this is one reason why I prefer to use base R packages.
$\endgroup$
2
$\begingroup$

In my experience, most of the changes create the usual computing/programming type of issues. Functions get deprecated, arguments are different, etc. For example, it already happened to me that code would stop working because some function required an extra argument. This can be annoying but the problem is obvious and usually not too difficult to resolve.

Some packages could be better maintained in that respect but the usual “rules” of free software apply: You have to see that it is produced by volunteers – often statisticians who have other duties and not full-time professional software developers – and if quality and dependability are important to you, you should avoid anything with a version number like 0.x and only use mature packages with an active community of users and developers.

I have never encountered an update that would result in insidious changes of statistical results (e.g. switching the default method in some function, changing the degree of freedom and p value while producing superficially similar results). I guess it must happen though, at least through bug corrections (but I have read somewhere that Microsoft actually added code in its new products to emulate some older bugs to avoid breaking compatibility with programs that depend on these bugs). Maybe, some maintainers can give us some insights into the way these things are handled for their packages.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.