5
$\begingroup$

Suppose that I have taken 8 machine learning algorithms which are used by researchers most frequently. I have applied these 8 machine learning algorithms over 8 datasets which are publicly available on internet.

I get results like: Random forest works well on 1 dataset. SVM performs better on 2 dataset. How can I conclude which machine learning algorithm among all performs best.

Thanks in advance..

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
10
$\begingroup$

For classification algorithms this would be a good start: Statistical Comparisons of Classifiers over Multiple Data Sets.

To summarize this excellent paper: perform a Friedman test to determine if there is any significant difference between the classifiers and follow-up with an appropriate post-hoc test if there is:

  • to compare all classifiers: Nemenyi test
  • to compare one with all others: Bonferroni-Dunn test

Both post-hoc tests can be visualized neatly in so-called critical difference diagrams.

Improve this answer
$\endgroup$
1
$\begingroup$

The notion of which Machine Learning algorithm is best is not universal, rather specific to the problem or the dataset you are dealing with.

In case of a single dataset or a problem, apply all learning algorithms and check the performance on out of sample data. Calculate Root mean Square errors between the predicted and actual values of out of sample data, and the algorithm with least RMSE will be the best only for that dataset.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Yes, the no free lunch theorem applies. No, it does not mean that you can't generalize anything beyond the data sets you have explicitly tested. Many data sets have a lot in common, even if they stem from entirely different domains. This allows us to generalize the expected performance of learning algorithms to some extent and is why some approaches tend to work well for a lot of practical problems (RF, SVM, NN, ...). $\endgroup$
    – Marc Claesen
    Commented Oct 29, 2014 at 7:23
  • $\begingroup$ Absolutely true. We can generalize anything beyond the dataset which we have tested only when we have some background knowledge about the current and the new dataset. In case of randomly taken dataset (which I inferred from the question) this does not hold true. Methods like RF which are ensemble tend to work well, as they are very flexible because they use multiple learning algorithms inside them. $\endgroup$
    – show_stopper
    Commented Oct 29, 2014 at 7:30
  • $\begingroup$ IMHO in order to generalize you need knowledge about both the data and the algorithms. The question is: how well does the heuristic behind the algorithm fit to the struture of the data? E.g. SVM focuses onto few boundary cases, but it will depend on the data (including the application) whether this is a sensible approach (if the single cases are too noisy, you may end up with a margin width that includes most of the cases - and then maybe other heuristics that focus less on single instances, or even give equal weight to all cases from the beginning are better) $\endgroup$
    – cbeleites
    Commented Oct 29, 2014 at 10:55
  • $\begingroup$ One more question is there any difference between machine learning and stastitical techniques. I have searched a lot some researchers say that there are some overlap some are saying there is no difference.Can you give some research paper which suggest that both are different from of classification. $\endgroup$
    – user28681
    Commented Oct 30, 2014 at 11:50
  • $\begingroup$ statistics vs machine learning is an interesting debate going on for sometimes now. In simple words as described in the tread bellow, machine learning is statistics minus any checking of models and assumptions stats.stackexchange.com/questions/6/… $\endgroup$
    – show_stopper
    Commented Oct 31, 2014 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.