# Who uses R with multicore, SNOW or CUDA package for resource intense computing?

Who of you in this forum uses ">R with the multicore, snow packages, or CUDA, so for advanced calculations that need more power than a workstation CPU? On which hardware do you compute these scripts? At home/work or do you have data center access somewhere?

The background of these questions is the following: I am currently writing my M.Sc. thesis about R and High Performance Computing and need a strong knowledge about who actually uses R. I read that R had 1 million users in 2008, but that's more or less the only user statistics I could find on this topic - so I hope for your answers!

Sincerely Heinrich

-
A possible related question, stats.stackexchange.com/questions/825/…. –  chl Nov 16 '10 at 8:33
Exact duplicate of stat.ethz.ch/pipermail/r-help/2010-November/259921.html –  Joshua Ulrich Nov 17 '10 at 17:03

I am a biologist who models the effects of inter-annual climatic variation on population dynamics of several migratory species. My datasets are very large (spatially intensive data) so I run my R code using multicore on Amazon EC2 servers. If my task is particularly resource intensive, I will choose a High Memory Quadruple Extra Large instance which comes with 26 CPU units, 8 cores, and 68G of RAM. In this case I usually run 4-6 scripts simultaneously, each of which is working through a fairly large data set. For smaller tasks, I choose servers with 4-6 cores and about 20 gigs of RAM.

I launch these instances (usually spot instances because they are cheaper but can terminate anytime the current rate exceeds what I have chosen to pay), run the script for several hours, and then terminate the instance once my script has finished. As for the machine image (Amazon Machine Image), I took someone elses Ubuntu install, updated R, installed my packages, and saved that as my private AMI on my S3 storage space.

My personal machine is a dualcore macbook pro and it has a hard time forking multicore calls. Feel free to email if you have other questions.

-
Can you pls tell what is the size of your data set. –  suncoolsu Nov 16 '10 at 19:28
Sure. The datasets I am currently working with are ~14 gigs –  Maiasaura Nov 17 '10 at 18:55

Since you ask, I am using the foreach package with the multicore backend. I use it to split an embarrassingly parallel workload across multiple cores on a single Nehalem box with lots of RAM. This works pretty well for the task at hand.

-
Thanks for your answer! Do you do the computation for your work/academic research or for own projects on a own PC? –  Heinrich Nov 16 '10 at 9:59
This is done in a commercial setting. For this task, I am using a single Intel box with 32GB of RAM and RAIDed disks (the main difficulty is lots of data, while the processing itself is not very computationally demanding.) –  NPE Nov 16 '10 at 11:01
Alright @aix, how often do you perform these calculations. Is you box running all the day or more idle? –  Heinrich Nov 16 '10 at 12:38
Quick question to @NPE: in what system do you store the data ? do you use a database back-end ? –  cafe876 Nov 26 '12 at 8:47

I work in academy and I'm using multicore for some heavy benchmarks of machine learning algorithms, mostly on our Opteron based Sun Constellation and some smaller clusters; those are also rather embarrassingly parallel problems so multicore's main role is to spread computation over node without multiplication of memory usage.

-