I've got a small list of companies that provide a platform for running R, python, or octave scripts on clusters built on top of amazon EC2. Are there other names I should add?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
migrated from stackoverflow.com Nov 4 '11 at 9:13
This question came from our site for professional and enthusiast programmers.
I've not used these firms, but these are just some of the options. However, setting up on EC2 is very simple. At least after you've numbed your head by banging it against the wall hard enough and often enough. It also increases your pain threshold. Kids these days have it so much easier. When I was getting started, we were doing fog computing: we hadn't the foggiest what we were computing.
Reminiscences aside... Tools and resources exist to bypass these vendors and get started on your own. JD Long has, with
Regarding Python, boto is a suite that allows for getting Python up and running easily on EC2. It's also pretty easy to get Python up and running on Google's AppEngine, if you're willing to consider an EC2 alternative. If you know Python, then there's really no need to hire a company to do the lifting for you, unless you are unfamiliar with scaling, sharding, load balancing, etc., at even a conceptual level. On the other hand, if you are contemplating spending a lot of money on such services, it's a good idea to become familiar with how to make your code efficient.
As for Octave on the cloud, I have no idea what's exists other than these three companies. Monkey Analytics used to offer it, but it seems they're gone. I would recommend avoiding Octave, and focus on Python or R.
One resource for making it easier to get set up is StarCluster. Again, no experience with them, but that may be a helpful route.
In all honesty, Ubuntu (or Windows) and EC2 are not that hard to learn. I wouldn't really recommend Windows for R, as there's not a lot of love among R developers and Windows. (NB: As far as I can tell there are no serious Mac OS X cloud services.) Once you have a remote desktop, you're in business. Learning how to scale is the next step.
Update 1: Other, more general, cloud managing services include RighstScale and Scalr.
Update 2: I want to emphasize that learning to set up your instances and clusters in the cloud for yourself is important. Among the benefits of hands-on work:
There are benefits to using a managed provider, such as a shorter learning curve, possibly better support for sharing resources among a group, and maybe some nice gizmos, but I can't speak to the benefits as I began using EC2 before any of these came to market.
There's also PiCloud for running Python code in parallel on EC2.
From their product page:
Did you try CloudStat before? Different from other clouds, CloudStat is meant only for R Language users. There is no more setup. you can just login and use up to 7.5 Gb RAM for free.
However, with free account, your analysis will be viewed and used by public. The option is pay $ 5 per month to make your analysis private.