0
$\begingroup$

I have developed a script using pythons scipy package to analyse a rather large model that I wish to solve, the model contains over 12gb of data, including over 500 parameters. Now running small simulations of about 0.5gb of data with 20 parameters can still take my computer a decent amount of time if given a reasonable number of iterations through the random forest classifier. Currently my script is only using one core, so I guess that could be the first step, to make the script multi-threaded. But this will not be enough given how complex the model is. I am willing to explore the use of a cluster based HPC solution but I am not sure how to go about this.

Any advice on the above would be appreciated.

Thanks AEA

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

scikit-learn's random forest implementation can use all available cores by setting the parameter n_jobs=-1

If you want a more industrial-strength tool, you use can try h2o 0xdata.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi Emre, what qualifies h2o as a more industrial tool? $\endgroup$
    – AEA
    Commented Jun 18, 2014 at 7:27
  • $\begingroup$ It can run on Hadoop and it has its own cluster manager, whereas sklearn only runs on one machine. 0xdata is made for production use, and is written in Java, so it will be faster than sklearn. $\endgroup$
    – Emre
    Commented Jun 18, 2014 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.