2
$\begingroup$

Speed isn't a concern for many statistics projects but sometimes it is, for example MCMC. Assume that hardware improvement is not an option (looking for relatively free solutions that produce orders of magnitude speed improvements over using an interpreted language). This post is inspired by the Highest voted question on Cross Validated that discusses using Python as statistics workbench.

In this answer, csgillespie points out that one reason to use python is the ability to interface with C without having to write cumbersome C code when you need speed (for example when doing MCMC). However, packages do exist that allow you to interface with C and Fortran directly from R.

Sometimes I need more speed. I do not know either C or Python or Fortran, so as a person who would like to learn a a tool that improves speed when I need it what are the pros and cons of using Python vs. calling C directly in R? Or pros and cons of other options. Preferably the answer should address the learning curve of learning the new tool in addition to its eventual usefulness after mastery.

The reason for this question: Currently Cross Validated maintains a list of resources for statistical computing. However, That list contains absolutely nothing about how to interface from R or python with compiled languages that can improve speed. Note I feel this question is most appropriate for crossvalidated and not stackoverflow because of the audience (the answer is intended to be for statisticians). However, if you do think this question is off topic please comment so that it can either be edditted for improvement or moved to an appropriate location.

Improve this question
$\endgroup$
4
  • $\begingroup$ When I had this problem I learned the rudimentary C++ necessary for the specific task and used Rcpp. While I am sure my implementation is not as efficient as could be, it was not that difficult to figure out a way that increased the speed over R by many times. $\endgroup$
    – Livid
    Commented Apr 9, 2015 at 23:09
  • 1
    $\begingroup$ A faster computer? $\endgroup$
    – wolfies
    Commented Apr 10, 2015 at 15:43
  • 1
    $\begingroup$ For those who voted this as off topic the help center implies this question is borderline. I can't see how "Python as a statistical workbench" is more on topic than this question? That was the intended flavor of my question, so if there are improvements to get this question to that level of question writing, please comment. $\endgroup$
    – WetlabStudent
    Commented Apr 10, 2015 at 15:54
  • 2
    $\begingroup$ On on-topic/off-topic: Call us inconsistent, but I think dividing lines have shifted since 2010 when the Python question was asked; I wasn't a member then but older questions are sometimes not of a kind that would (I think) now be accepted. It's a repeatedly difficult and contentious issue how far software-related questions are welcome here. My personal perception is that the bottom line is that no forum wants questions centred on simple syntax errors or "please give me the code"; that is where most exclusion criteria grow from. But this question is quite different from those. $\endgroup$
    – Nick Cox
    Commented Apr 10, 2015 at 17:41

2 Answers 2

Reset to default
2
$\begingroup$

The answer you point out is an extremely specialized problem: doing MCMC wherein each iteration needs to solve 5 ODEs. For more mainstream MCMC problems, you have access to JAGS (rjags) and Stan (rstan). For extremely specialized problems, R has the Rcpp and inline packages that you might find useful.

I'm not sure how freely Stan lets you do arbitrary C++ code, but that may also be a way to accomplish something similar to what the referenced answer does: use R where it's strong, call Stan where you need MCMC speed, and within Stan do customized evaluation in C++.

I also think that smarter algorithms can make a huge difference in some cases. For example, LaplacesDemon has some cutting-edge samplers that allow it to often be competitive with compiled packages like JAGS or Stan, even though it's entirely written in R. That wouldn't help as much in the referenced answer because so much of the time was spent solving ODE's, not doing MCMC, but even then an algorithm that converges 10x faster needs 1/10 the evaluations.

Are you inquiring about a general need for speed, or do you have some specific examples? It seems a bit harsh to learn Python in order to learn and use C.

Improve this answer
$\endgroup$
1
$\begingroup$

Are you planning on writing the C or Python that you need? Or are you having an expert in the respective language do it?

If the former, choose Python as it will take you one tenth the time and effort to get functional with as C. Well-written, C is the faster language as it is is only one step removed from the processor instruction set.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.