7
$\begingroup$

Is it a good or a bad practice to use R packages from CRAN for research? I'm talking about the common packages like: simple models for regression, estimation, econometrics.
Most of them use function that can be written easily on your own.

My yes arguments:

  • Time to focus on the main part of the research
  • The community is a good quality control
  • R offers a lot of modern methods
  • Some models are too complicated to write them on your own

My con arguments:

  • Not every package of R has been created in an academic environment
  • There could be bugs that influence the outcome and I do not know it
  • Most models can be written in a short time period from scratch

How can we use open source without risking failures in the outcome? Are there certain quality indicators for packages in general and for R?

$\endgroup$
13
  • 1
    $\begingroup$ That statement's a good point. $\endgroup$
    – Bene
    Commented Aug 3, 2013 at 10:42
  • 8
    $\begingroup$ I tend to think that writing something from scratch when a suitable library is available is almost always a bad idea. All arguments you could invoke against open-source software (potential for bugs, no guarantees they were properly tested, not necessarily developed by statisticians/academics/economists/software engineers…) are even more valid against home-cooked solutions. $\endgroup$
    – Gala
    Commented Aug 3, 2013 at 11:07
  • 2
    $\begingroup$ Relevant questions: R vs SAS, why is SAS prefered by private companies? Is the R language reliable in the field of economics? $\endgroup$
    – Gala
    Commented Aug 3, 2013 at 11:11
  • 3
    $\begingroup$ One is forced to write publications properly. The software used is rarely under scrutiny. $\endgroup$ Commented Aug 3, 2013 at 11:14
  • 3
    $\begingroup$ @Gavin Simpson Your comments about commercial software are accurate in so far as compiled code is inaccessible with seeing source code too. But commercial software often includes code in its own language that can be inspected e.g. MATLAB, Stata. Also, in principle, everyone can look deep, deep inside R but only some tiny fraction of users can find bugs in really fundamental code. I suggest that this often sung song exaggerates the practical differences. There is a difference of principle, but the practical differences are not quite as great as often stated. $\endgroup$
    – Nick Cox
    Commented Aug 3, 2013 at 20:23

1 Answer 1

13
$\begingroup$

I don't consider this an R specific question. The real question is: can you trust other people's code? Or, taking the other perspective: do you think you can do better? (in the time you are willing/able to spend)

Whether the software is open or closed source, does not really matter. The trustworthiness of open source compared to commercial software is a topic of much debate. Arguments exist in favor of either side of the discussion. Nobody can refute that both can (and do) contain bugs of various nature, some more insidious than others.

TL;DR: in my opinion: yes, it's fine to use existing packages if they cover your needs.


My motto, in general, is to try to implement stuff myself if a variety of the following are true:

  1. I know exactly what I want and how to do it (this is far from trivial for many practical models).
  2. It is worth the time required to implement and test my own code compared to what already exists. In other words: will I use it often or just once?
  3. What already exists does not offer all the functionality I need.
  4. I have a good reason not to trust existing implementations (such as unexpected behaviour when using it).

Personally, I like, use and develop a lot of open source as I believe that to be important, especially in an academic context. In practice, many people only consider using a given algorithm if an implementation is available. Nobody benefits from a large set of implementations of the same thing. It is far better to dispose of one efficient, thoroughly tested and verified implementation of a given method.

People like to believe that if I do it myself, I know it's done properly. Practice contradicts this on regular basis.

Not every package of R has been created in an academic environment

I don't know why you feel that packages created by academics are superior. Sure, the method itself may be more complex/novel, but all bets are off regarding implementation. Researchers are not necessarily the best programmers (in fact, given that that is usually not their specialty I would say the opposite is more likely).

In practice, researchers regularly hack solutions together, sometimes without thorough testing. Naturally, this can lead to all kinds of bad results (most notably silent fails). Personally, I believe one of the reasons for this behaviour is the fact that software output is undervalued in research settings. Some researchers simply rush towards the next publication. Publication of results matter, the software used to get them doesn't really. This leads to software that has not been tested properly when people reinvent the wheel for a single use.

How can we use open source without risking failures in the outcome?

You can't. This also applies to commercial software and software you write yourself with lots of care and love. If you happen to find a way to ensure software has no bugs or caveats, you should instantly make an appointment with Bill Gates (or better yet, tell me about it).

The only solution is to critically evaluate results every step of the way. Never trust software unconditionally, no matter what software it is.

Are there certain quality indicators for packages in general and for R?

Usually, packages that get made publicly available, for example on CRAN, have been subjected to rigorous tests, especially the ones that get used often. People will think twice before making untested garbage publicly available when their credibility is at stake. The number of citations can be considered a quality indicator too in addition to an active developing community.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.