I have R-scripts for reading large amounts of csv data from different files and then perform machine learning tasks such as svm for classification.
Are there any libraries for making use of multiple cores on the server for R.
or
What is most suitable way to achieve that?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ I just don't see how the fact that importing data and running SVM has any relevance to the question. That's why I think it's more of an SO question. But I could see Xrefs as being a good long-term solution since it is R... $\endgroup$– ShaneCommented Jul 27, 2010 at 17:20
-
3$\begingroup$ I have no problem with this sort of Q&A here. R isn't such a mainstream language (like Python or Java) that a quant would naturally say, "Oh this is a general programming question so I should go to StackOverflow or similar and ask this or look there for solutions". Actually it is more a question for an R mailing list or group site. To serve those budding analysts who want to learn R we should be glad to have an answer here as well. $\endgroup$– PaulCommented Aug 1, 2010 at 10:32
-
2$\begingroup$ Vote to keep open; very relevant to statisticians because the ways in which our problems can or can not be broken down into parallel streams is of relevance to the question being asked. $\endgroup$– russellpierceCommented Aug 4, 2010 at 14:40
-
$\begingroup$ @chl: Thanks for bumping this up. In fact, I checked out all the non-commercial references from this thread shortly after it appeared but couldn't find anything that works on Win 7 x64. $\endgroup$– whuber ♦Commented Oct 14, 2010 at 18:07
-
1$\begingroup$ whuber, the solution I present works with win 7 and is non commercial (read the post I linked to for details). It is bundled with a commercial environment but it can be separated from it (as my post shows how). And the code itself is GPL... $\endgroup$– Tal GaliliCommented Oct 16, 2010 at 8:44
Comments disabled on deleted / locked posts / reviews
|
5 Answers
$\begingroup$
$\endgroup$
2
If it's on Linux, then the most straight-forward is multicore. Beyond that, I suggest having a look at MPI (especially with the snow package).
More generally, have a look at:
Lastly, I recommend using the foreach package to abstract away the parallel backend in your code. That will make it more useful in the long run.
-
$\begingroup$ I mainly use multicore, still I like snowfall more than snow and Rmpi for its fault tolerance and clean interface. $\endgroup$– user88Commented Jul 27, 2010 at 18:25
-
$\begingroup$ @mbq +1 for snowfall- abstracts snow even further and makes parallel computing with R pretty simple. $\endgroup$– SharpieCommented Jul 31, 2010 at 18:14
$\begingroup$
$\endgroup$
If you are using GNU/Linux previous answers by Shane and Dirk are great.
If you need a solution for windows, there is one in this post:
Parallel Multicore Processing with R (on Windows)
Although the package is not yet on CRAN. it can be downloaded from that link.
answered Jul 30, 2010 at 16:58
$\begingroup$
$\endgroup$
2
Shane is correct. Both multicore and Rmpi are winners.
Slightly broader coverage of the topic is in the CRAN Task View on High-Performance Computing. This also links to a fairly recent survey article on Parallel Computing with R from JSS.
Lastly, a few hands-on examples and tips are in the Intro to HPC with R tutorial I give once in a while -- see my presentations page for the most recent copy from last week at useR.
answered Jul 27, 2010 at 17:12
-
$\begingroup$ Well, mutexes needed. As I commented on your answer, I only saw the first (raw) version and figured well, I may expand on mc and Rmpi. And then you did and I look like a copycat. Such is life. $\endgroup$ Commented Jul 27, 2010 at 17:15
-
$\begingroup$ On the other hand, my answer is derived from reading your paper/presentation in the past. So I guess I'm copying you as well. $\endgroup$– ShaneCommented Jul 27, 2010 at 17:16
$\begingroup$
$\endgroup$
2
I noticed that the previous answers lack some general HPC considerations.
First of all, neither of those packages will enable you to run one SVM in parallel. So what you can speed up is parameter optimization or cross-validation, still you must write your own functions for that. Or of course you may run the job for different datasets in parallel, if it is a case.
The second issue is memory; if you want to spread calculation over a few physical computers, there is no free lunch and you must copy the data -- here you must consider if it makes sense to predistribute a copy of data across computers to save some communication. On the other hand if you wish to use multiple cores on one computer, than the multicore is especially appropriate because it enables all child processes to access the memory of the parent process, so you can save some time and a lot of memory space.
-
1$\begingroup$ +1 Great point about how this doesn't deal with splitting up the cross-validation. $\endgroup$– ShaneCommented Jul 27, 2010 at 18:45
-
$\begingroup$ Incidentally, there has been some recent work (2013) in enabling HPC for individual SVMs by dCSE (hector.ac.uk/cse/distributedcse/reports/sprint03/…). There is a package
sprintfor R with a functionpsvm, but they are a little behind on keeping up with the R 3.0 changes and new CRAN submission guidelines, so the current download is neither available on CRAN or fully compatible with R 3.0. Your mileage may vary. $\endgroup$ Commented May 21, 2014 at 5:48
$\begingroup$
$\endgroup$
2
Both Shane and Dirk's responses are spot on.
Nevertheless, you might wanna take a look at a commercial version of R, called Revolution R which is built to deal with big datasets and run on multiple cores. This software is free for academics (which might be your case, I dont know)
answered Jul 27, 2010 at 17:18
-
5$\begingroup$ I disagree somewhat. Revolution does a great sales job in getting mindshare (as evidenced by your post) but as of right now there is very little in the product you would not already get with the normal R (at least on Linux). Intel MKL, sure, but you can get Goto Blas. On Windows, they offer doSMP which helps as multicore cannot be built there. $\endgroup$ Commented Jul 27, 2010 at 17:42
-
2$\begingroup$ But, of course, doSMP is exactly what the OP would be looking for if they were working in a Windows environment. $\endgroup$ Commented Aug 4, 2010 at 14:44