28
$\begingroup$

I have R-scripts for reading large amounts of csv data from different files and then perform machine learning tasks such as svm for classification.
Are there any libraries for making use of multiple cores on the server for R.
or
What is most suitable way to achieve that?

$\endgroup$
5
  • $\begingroup$ I just don't see how the fact that importing data and running SVM has any relevance to the question. That's why I think it's more of an SO question. But I could see Xrefs as being a good long-term solution since it is R... $\endgroup$ – Shane Jul 27 '10 at 17:20
  • 3
    $\begingroup$ I have no problem with this sort of Q&A here. R isn't such a mainstream language (like Python or Java) that a quant would naturally say, "Oh this is a general programming question so I should go to StackOverflow or similar and ask this or look there for solutions". Actually it is more a question for an R mailing list or group site. To serve those budding analysts who want to learn R we should be glad to have an answer here as well. $\endgroup$ – Paul Aug 1 '10 at 10:32
  • 2
    $\begingroup$ Vote to keep open; very relevant to statisticians because the ways in which our problems can or can not be broken down into parallel streams is of relevance to the question being asked. $\endgroup$ – russellpierce Aug 4 '10 at 14:40
  • $\begingroup$ @chl: Thanks for bumping this up. In fact, I checked out all the non-commercial references from this thread shortly after it appeared but couldn't find anything that works on Win 7 x64. $\endgroup$ – whuber Oct 14 '10 at 18:07
  • 1
    $\begingroup$ whuber, the solution I present works with win 7 and is non commercial (read the post I linked to for details). It is bundled with a commercial environment but it can be separated from it (as my post shows how). And the code itself is GPL... $\endgroup$ – Tal Galili Oct 16 '10 at 8:44
16
$\begingroup$

If it's on Linux, then the most straight-forward is multicore. Beyond that, I suggest having a look at MPI (especially with the snow package).

More generally, have a look at:

  1. The High-Performance Computing view on CRAN.
  2. "State of the Art in Parallel Computing with R"

Lastly, I recommend using the foreach package to abstract away the parallel backend in your code. That will make it more useful in the long run.

$\endgroup$
2
  • $\begingroup$ I mainly use multicore, still I like snowfall more than snow and Rmpi for its fault tolerance and clean interface. $\endgroup$ – user88 Jul 27 '10 at 18:25
  • $\begingroup$ @mbq +1 for snowfall- abstracts snow even further and makes parallel computing with R pretty simple. $\endgroup$ – Sharpie Jul 31 '10 at 18:14
6
$\begingroup$

If you are using GNU/Linux previous answers by Shane and Dirk are great.

If you need a solution for windows, there is one in this post:

Parallel Multicore Processing with R (on Windows)

Although the package is not yet on CRAN. it can be downloaded from that link.

$\endgroup$
5
$\begingroup$

Shane is correct. Both multicore and Rmpi are winners.

Slightly broader coverage of the topic is in the CRAN Task View on High-Performance Computing. This also links to a fairly recent survey article on Parallel Computing with R from JSS.

Lastly, a few hands-on examples and tips are in the Intro to HPC with R tutorial I give once in a while -- see my presentations page for the most recent copy from last week at useR.

$\endgroup$
2
  • $\begingroup$ Well, mutexes needed. As I commented on your answer, I only saw the first (raw) version and figured well, I may expand on mc and Rmpi. And then you did and I look like a copycat. Such is life. $\endgroup$ – Dirk Eddelbuettel Jul 27 '10 at 17:15
  • $\begingroup$ On the other hand, my answer is derived from reading your paper/presentation in the past. So I guess I'm copying you as well. $\endgroup$ – Shane Jul 27 '10 at 17:16
5
$\begingroup$

I noticed that the previous answers lack some general HPC considerations.
First of all, neither of those packages will enable you to run one SVM in parallel. So what you can speed up is parameter optimization or cross-validation, still you must write your own functions for that. Or of course you may run the job for different datasets in parallel, if it is a case.
The second issue is memory; if you want to spread calculation over a few physical computers, there is no free lunch and you must copy the data -- here you must consider if it makes sense to predistribute a copy of data across computers to save some communication. On the other hand if you wish to use multiple cores on one computer, than the multicore is especially appropriate because it enables all child processes to access the memory of the parent process, so you can save some time and a lot of memory space.

$\endgroup$
2
  • 1
    $\begingroup$ +1 Great point about how this doesn't deal with splitting up the cross-validation. $\endgroup$ – Shane Jul 27 '10 at 18:45
  • $\begingroup$ Incidentally, there has been some recent work (2013) in enabling HPC for individual SVMs by dCSE (hector.ac.uk/cse/distributedcse/reports/sprint03/…). There is a package sprint for R with a function psvm, but they are a little behind on keeping up with the R 3.0 changes and new CRAN submission guidelines, so the current download is neither available on CRAN or fully compatible with R 3.0. Your mileage may vary. $\endgroup$ – bright-star May 21 '14 at 5:48
3
$\begingroup$

Both Shane and Dirk's responses are spot on.

Nevertheless, you might wanna take a look at a commercial version of R, called Revolution R which is built to deal with big datasets and run on multiple cores. This software is free for academics (which might be your case, I dont know)

$\endgroup$
2
  • 5
    $\begingroup$ I disagree somewhat. Revolution does a great sales job in getting mindshare (as evidenced by your post) but as of right now there is very little in the product you would not already get with the normal R (at least on Linux). Intel MKL, sure, but you can get Goto Blas. On Windows, they offer doSMP which helps as multicore cannot be built there. $\endgroup$ – Dirk Eddelbuettel Jul 27 '10 at 17:42
  • 2
    $\begingroup$ But, of course, doSMP is exactly what the OP would be looking for if they were working in a Windows environment. $\endgroup$ – russellpierce Aug 4 '10 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.