Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've got a small list of companies that provide a platform for running R, python, or octave scripts on clusters built on top of amazon EC2. Are there other names I should add?

  1. Cloudnumbers
  2. Opani
  3. crdata
share|improve this question

migrated from Nov 4 '11 at 9:13

This question came from our site for professional and enthusiast programmers.

<comments about closing at SO removed> – mbq Nov 4 '11 at 10:43
Yay, a new home for this question!!! :) – Iterator Nov 4 '11 at 11:04
Of course, I should have posted this to crossvalidated first. Thanks for moving it! – Zach Nov 4 '11 at 13:29
see – Antoine Aug 24 '15 at 20:00
up vote 15 down vote accepted

I've not used these firms, but these are just some of the options. However, setting up on EC2 is very simple. At least after you've numbed your head by banging it against the wall hard enough and often enough. It also increases your pain threshold. Kids these days have it so much easier. When I was getting started, we were doing fog computing: we hadn't the foggiest what we were computing.

Reminiscences aside... Tools and resources exist to bypass these vendors and get started on your own. JD Long has, with segue, made it easier to start using R on EMR.

Regarding Python, boto is a suite that allows for getting Python up and running easily on EC2. It's also pretty easy to get Python up and running on Google's AppEngine, if you're willing to consider an EC2 alternative. If you know Python, then there's really no need to hire a company to do the lifting for you, unless you are unfamiliar with scaling, sharding, load balancing, etc., at even a conceptual level. On the other hand, if you are contemplating spending a lot of money on such services, it's a good idea to become familiar with how to make your code efficient.

As for Octave on the cloud, I have no idea what's exists other than these three companies. Monkey Analytics used to offer it, but it seems they're gone. I would recommend avoiding Octave, and focus on Python or R.

One resource for making it easier to get set up is StarCluster. Again, no experience with them, but that may be a helpful route.

In all honesty, Ubuntu (or Windows) and EC2 are not that hard to learn. I wouldn't really recommend Windows for R, as there's not a lot of love among R developers and Windows. (NB: As far as I can tell there are no serious Mac OS X cloud services.) Once you have a remote desktop, you're in business. Learning how to scale is the next step.

Update 1: Other, more general, cloud managing services include RighstScale and Scalr.

Update 2: I want to emphasize that learning to set up your instances and clusters in the cloud for yourself is important. Among the benefits of hands-on work:

  • Learn how to manage a mixture of resources (multiple instances, multiple instance types, lots of HDs, different availability zones or regions, various monitoring tools, and more)
  • Tinker with GPUs if you wish (check out gputools)
  • You can more easily update or roll-back your selection of packages
  • You may be able to get much lower costs by using either spot instances or reserved instances.
  • You can try out different R GUIs or IDEs, which might not be an option for the cloud vendors.

There are benefits to using a managed provider, such as a shorter learning curve, possibly better support for sharing resources among a group, and maybe some nice gizmos, but I can't speak to the benefits as I began using EC2 before any of these came to market.

share|improve this answer

There's also PiCloud for running Python code in parallel on EC2.

From their product page:

PiCloud is a cloud-computing platform that integrates into the Python Programming Language. It enables you to leverage the computing power of Amazon Web Services without having to manage, maintain, or configure virtual servers.

PiCloud integrates seamlessly into your existing code base through a custom Python library, cloud. To offload the execution of a function to our servers, all you must do is pass your desired function into the cloud library. PiCloud will run the function on its high-performance cluster. As you run more functions, our cluster auto-scales to meet your computational needs. Getting on the cloud has never been this easy!

share|improve this answer
I'm rather fond of PiCloud - I've found them remarkably easy to get up and running. – Fomite Nov 7 '11 at 5:02

Did you try CloudStat before? Different from other clouds, CloudStat is meant only for R Language users. There is no more setup. you can just login and use up to 7.5 Gb RAM for free.

However, with free account, your analysis will be viewed and used by public. The option is pay $ 5 per month to make your analysis private.

share|improve this answer

One cloud computing platform that I can recommend are Backbone who provide a secure productivity enhancing cloud computing platform.

share|improve this answer
Do they have an R-specific offering? I'm not looking for a general cloud-computing provider, I'm looking for one that provides pre-configured R (preferably RStudio) servers. – Zach May 27 '14 at 18:25
see – Antoine Aug 24 '15 at 19:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.