parallelism in data mining softwares

Question

I'm working on a data set for order prediction/classification and a close deadline upcoming. Fortunately, my university has a super-computer with restricted access. I was thinking of using a few nodes (each node is a 16-core processor, 2.3 GHz each and 32 GB of memory running on linux) to overcome the time limitation I'm facing.

Basic software that I have in mind is WEKA, RapidMiner, MATLAB and maybe R. But as it's my first confrontation to parallel data mining, I need some help on this. I think the software needs to have a parallel nature in order to get most out of multi-core processors and using a lot of cores doesn't make any significant speed-up on serial programs.

Best result of my searches was WEKA-parallel that is a modification of WEKA on distributed systems.

At last I'm open to any suggestion on getting most out of this computational power specially using WEKA. New software, new method, anything...

Answer 1

I say you are looking at the problem upside down. Just to save analysis time if you try to jump into parallising the solution without ever attempting a simple single core solution you are in for a whole lot of pain.

I suggest you get at least a basic non-parallel solution coded up, then debug it and after that start thinking about parallalization.

Explicitly parallalising codes isn't a trivial task. One easier way may be task parallalization: say, run various training cases with one parameter on each machine. This doesn't even need the code / software to be parallel aware. You could have a scripting language wrapper in something like perl or bash spawn off each training case to a different node.

Also, it would help if you told us the size and nature of your dataset. Is it really pushing the limits of a single 32 Gig node?

Parallel processing is a good way to ease computing constraints but if you are attempting to reduce your human analytical load the short cut ain't going to work.

e.g. If you don't have enough time to meet the deadline for your Physics-101 homework applying advanced String Theory is not going to help make things any faster.

Answer 2

You're correct in that code needs to be written in a specific way to benefit from having multiple cores/processors. It's sometimes non-trivial to develop a parallel version of an algorithm that actually benefits from multiple cores, so it might be hard to get meaningful speedups in an algorithm you've developed yourself. That said, there are definitely packages around that have parallel versions of some machine learning algorithms.

Another use-case, which makes a lot of sense for machine learning applications, is to run multiple copies of single-core-friendly code. For example, if you're doing 10-fold cross validation, you could farm each fold out to a separate core and get the results (roughly) 10x as fast. Similarly, if you're doing a search for some parameters, you could run several copies, each starting from different locations, and then compare the results later.

This can be done by hand (just start 10 copies of the program), but a lot of packages (including WEKA, R, and Matlab) have some infrastructure to make it easier.

For WEKA, this thread has some suggestions, but the gist is that you can have one RemoteEngine client running per core and then assign the work that way. The GUI has some built-in multicore support too.

R has several options (see this StackOverflow thread and links therein). I haven't played with much personally, but I know a few people who are very happy with Revolutions.

Some Matlab functions have built-in support for multicore/multiprocessors (you'll have to look at the docs; it's a little bit random so far). The Parallel Computing toolbox has some nifty functionality whereby you can make a for-loop run in parallel. You basically do something like

matlabpool open;
result = nan(N,1);
parfor ii = 1:N
  result(ii) = some_function(ii);
end

This automatically distributes calls to some_function across cores and collects the results for you. There are---of course---some caveats, but it's pretty close to that simple, and it gracefully degrades to running sequentially if run on a computer without the toolbox. I suspect that you could do something similar with the right packages in R, but I'm not sure.

Answer 3

One tool that might work for this is ML-Flex (http://mlflex.sourceforge.net).