I've got a small list of companies that provide a platform for running R, python, or octave scripts on clusters built on top of amazon EC2. Are there other names I should add?
closed as off-topic by gung - Reinstate Monica♦ Jul 30 at 20:36
This question appears to be off-topic. The users who voted to close gave this specific reason:
- "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – gung - Reinstate Monica
migrated from stackoverflow.com Nov 4 '11 at 9:13
This question came from our site for professional and enthusiast programmers.
I've not used these firms, but these are just some of the options. However, setting up on EC2 is very simple. At least after you've numbed your head by banging it against the wall hard enough and often enough. It also increases your pain threshold. Kids these days have it so much easier. When I was getting started, we were doing fog computing: we hadn't the foggiest what we were computing.
Reminiscences aside... Tools and resources exist to bypass these vendors and get started on your own. JD Long has, with
segue, made it easier to start using R on EMR.
Regarding Python, boto is a suite that allows for getting Python up and running easily on EC2. It's also pretty easy to get Python up and running on Google's AppEngine, if you're willing to consider an EC2 alternative. If you know Python, then there's really no need to hire a company to do the lifting for you, unless you are unfamiliar with scaling, sharding, load balancing, etc., at even a conceptual level. On the other hand, if you are contemplating spending a lot of money on such services, it's a good idea to become familiar with how to make your code efficient.
As for Octave on the cloud, I have no idea what's exists other than these three companies. Monkey Analytics used to offer it, but it seems they're gone. I would recommend avoiding Octave, and focus on Python or R.
One resource for making it easier to get set up is StarCluster. Again, no experience with them, but that may be a helpful route.
In all honesty, Ubuntu (or Windows) and EC2 are not that hard to learn. I wouldn't really recommend Windows for R, as there's not a lot of love among R developers and Windows. (NB: As far as I can tell there are no serious Mac OS X cloud services.) Once you have a remote desktop, you're in business. Learning how to scale is the next step.
Update 1: Other, more general, cloud managing services include RighstScale and Scalr.
Update 2: I want to emphasize that learning to set up your instances and clusters in the cloud for yourself is important. Among the benefits of hands-on work:
- Learn how to manage a mixture of resources (multiple instances, multiple instance types, lots of HDs, different availability zones or regions, various monitoring tools, and more)
- Tinker with GPUs if you wish (check out
- You can more easily update or roll-back your selection of packages
- You may be able to get much lower costs by using either spot instances or reserved instances.
- You can try out different R GUIs or IDEs, which might not be an option for the cloud vendors.
There are benefits to using a managed provider, such as a shorter learning curve, possibly better support for sharing resources among a group, and maybe some nice gizmos, but I can't speak to the benefits as I began using EC2 before any of these came to market.
There's also PiCloud for running Python code in parallel on EC2.
From their product page:
PiCloud is a cloud-computing platform that integrates into the Python Programming Language. It enables you to leverage the computing power of Amazon Web Services without having to manage, maintain, or configure virtual servers.
PiCloud integrates seamlessly into your existing code base through a custom Python library, cloud. To offload the execution of a function to our servers, all you must do is pass your desired function into the cloud library. PiCloud will run the function on its high-performance cluster. As you run more functions, our cluster auto-scales to meet your computational needs. Getting on the cloud has never been this easy!
Did you try CloudStat before? Different from other clouds, CloudStat is meant only for R Language users. There is no more setup. you can just login and use up to 7.5 Gb RAM for free.
However, with free account, your analysis will be viewed and used by public. The option is pay $ 5 per month to make your analysis private.