|
I'm looking for a good Java library implementing several clustering algorithms. I'll have to cluster some programs execution traces and I still don't know which algorithms I am going to need, so I'd like to use a library providing lot of them and that makes it easy to swap algorithms. So far I had a look at Weka but I don't know whether there is a more complete library available I'm missing. |
|||||||
|
|
Weka is very lacking and slow in clustering. KNIME and RapidMiner are mostly frontends to Weka when it comes to algorithms. You should have a look at ELKI, it has a lot more in clustering and outlier detection than Weka, and usually is a lot faster. There are quite some algorithms that I'm not aware of other implementations being publicly available except in ELKI. Just compare OPTICS in Weka and ELKI and you'll see the difference in the results and runtime. The only thing is: technically it's not a library. It's more of a standalone application, designed to develop new algorithms in the frameworks. But it's not a "feed a double[][] array in, execute magic(), get magic output result out" type of library. Data mining and clustering is more of an explorative process than something you can automate - it's the machine learning side that you can automate to some extend (most as in: once you have a good classifier, it will classify new data, too). But say, you analyze a data set one day, you find some structure. Over the next year, you data set grew to twice the size and the structure has disappeared, until you change some parameters, or even the algorithm. If you then find the same structure again, then you probably are on to some real structure in your data, at least, and not some random data artifact. :-) Mahout has some of the standard stuff, their focus is on scalability, not so much on flexibility. Some people use it on small data sets, too, but that doesn't make a lot of sense IMHO. It also is coming more from a classification side, like Weka, in my opinion. Data exploration in big data also is a lot harder. With machine learning, more data usually just makes your predictions better (if you can still process it all, which is where Hadoop and Mahout come in). But say you run a cluster analysis on a big data set, and you get out 1000 clusters - what are you going to do with them? How do you analyze them if they make any sense? You cannot really measure the quality of a clustering result without either comparing it with previous knowledge (at which point you did not gain anything except a shiny benchmark number, because you already know the "truth") or by manually inspecting the resulting cluster. |
|||
|
