10
votes
$\begingroup$

I've got R running on amazon EC2, using a modified version of the bioconductor AMI. Currently, I am using putty to ssh into my server, starting R from the command line, and then copying and pasting my script from notepad++ into my putty session.

The thing is, I hate cut and pasting. It feels stone-age and I occasionally get weird buffering issues that screw up my code. I can't use RStudio, because it doesn't support multicore, which I heavily depend on.

What's the more elegant way to do this?

/Edit: Thanks for all the great suggestions. For now, I've switched over to using foreach with the doRedis backend, which works great on my Mac, my PC, and on amazon through RStudio. This switch was pretty easy once I learned how to write a function that emulates "lapply" using "foreach." (Also, doRedis is awesome!)

$\endgroup$
3

6 Answers 6

Reset to default
12
votes
$\begingroup$

I can think of a few ways. I've done this quite a bit and here are the ways I found most useful:

  1. Emacs Daemon mode. ssh into the EC2 instance with the -X switch so it forwards X windows back to your remove machine. Using daemon mode will ensure that you don't lose state if your connection times out or drops
  2. Instead of using the multicore package, use a different parallel backend with the foreach package. That way you can use RStudio, which is fantastic. Foreach is great because you can test your code in non-parallel, then switch to parallel mode by simply changing your backend (1 or 2 lines of code). I recommend the doRedis backend. You're in the cloud, might as well fire up multiple machines!
$\endgroup$
3
  • $\begingroup$ +1 for Emacs Daemon mode. I've been doing quite a bit of ssh lately, and this looks really useful. $\endgroup$
    – richiemorrisroe
    Jun 1, 2011 at 22:55
  • $\begingroup$ Is there an easy way to emulate lapply using foreach? I've written a lot of code that leverages apply, and I like the multicore package because I can simple replace lapply with mclapply. Is there a 'foreachlapply,' or am I going to have to re-write a lot of code? Thanks! $\endgroup$
    – Zach
    Jun 2, 2011 at 0:43
  • 1
    $\begingroup$ Well, doRedis can only do redis stuff; huge input is not the only reason for HPC calculations. $\endgroup$
    – user88
    Jun 2, 2011 at 8:31
13
votes
$\begingroup$

The most convenient way is just to install VNC server and some light environment like XFCE and make yourself a virtual session that you can use from wherever you want (it persists disconnects), i.e. something like this: enter image description here

Additional goodies are that you can use your local clipboard in the virtual desktop and see R plots way faster than via X11 forwarding or copying image files.

It takes some effort to setup everything right (X init, ssh tunnel), but the internet is full of tutorials how to that.

$\endgroup$
2
  • 1
    $\begingroup$ Even better is NX, which has generally much improved performance characteristics. $\endgroup$
    – scw
    Jun 2, 2011 at 10:42
  • $\begingroup$ Could you like to a tutorial you like, or should I just look at the the top couple results on google? $\endgroup$
    – Zach
    Jun 9, 2011 at 16:58
3
votes
$\begingroup$

I don't know how Amazon EC2 works, so maybe my simple solutions don't work. But I normally use scp or sftp (through WinSCP if I'm on Windows) or git.

$\endgroup$
3
votes
$\begingroup$

I'd use rsync to push the scripts and data files to the server, then "nohup Rscript myscript.R > output.out &" to run things and when finished, rsync to pull the results.

$\endgroup$
1
  • 2
    $\begingroup$ screen or tmux are better than nohup -- they also detach the script so it won't be killed by logout, but allow to reattach the session and start over, even from other client computer. tmux can be even used as a kind of text-mode window manager. $\endgroup$
    – user88
    Jun 2, 2011 at 8:28
0
votes
$\begingroup$

VIM + tmux + VIM Slime. You get the greatest text editor and ability to send code from editor to R command line (just like in Rstudio).

$\endgroup$
0
votes
$\begingroup$

I use R Studio on EC2 all the time thanks to the AMIs created by Louis Aslett. You don't have to know any SSH or anything (other than R, of course). You just need an EC2 account. As mentioned in one of the other answers, R Studio does support parallel computing, via the foreach package for instance. This really enables harnessing the power of EC2. By using a compute-optimized instance (32 cores), I was able to significantly cut down training time for my ML models at almost no cost (a few bucks an hour).

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.