Sweeping across multiple classifiers and choosing the best?

Question

I'm using Weka to perform classification, clustering, and some regression on a few large data sets. I'm currently trying out all the classifiers (decision tree, SVM, naive bayes, etc.).

Is there an automated way (in Weka or other machine learning toolkit) to sweep through all the available classifier algorithms to find the one that produces the best cross-validated accuracy or other metric? I'm not talking about boosting; rather, I'm looking to just choose the best classifier using a given data set.

I'd like to find the best clustering algorithm, too, for my other clustering problem; perhaps finding the lowest sum-of-squared-error?

Answer 1

I would suggest a different approach. Instead of sweeping across all possible classifiers, stop and think about your problem. How does your feature space look like? For the case of binary classification, are there two large clusters with some boundary, or is your feature space "segmented" and contains many clusters?

In the former case, an SVM would be a good choice to separate the two clusters (with the right choice of kernel), in the latter a decision tree which splits the feature space into areas would probably be a better choice. Another issue is interpretability, do you need some sort of report or methodology for classification, or simply a prediction result? Decision tree can provide you with a methodology you can follow, enabling you to debug and check if you are overfitting. From my personal experience, understanding your dataset is at least as important as the choice of algorithm.

Answer 2

I did what you asked about, the best thing is, by using WEKA Java library .. in which you can choose all the classifiers that you want to test your data with them and then your application will start looping across them, so you can get the one that produce the highest F1, accuracy etc. value ..