# Comparing and evaluating various machine learning methods

I am not expert in this area so please bare with me. Is it possible to somehow evaluate the success rate of machine learning algorithm/methods. I suppose it could be done this way: Give a various ML one dataset and then check which one achieves the best score. This assumes that the correct outcome is already know and it will be used as a reference during comparison with ML. The problems that comes to my mind are:

1. various ML need to have various input formats, but this could be solved using some text preprocessing etc. (I do not know this for sure I'm just thinking aloud)
2. some ML are primary predetermined for specific tasks, so it would be best to compare "similar" families of ML algorithms?

Is there any study or even better the whole framework for this purpose? I am starting with ML and would like to try several algorithms and compare their results performance etc. Something practical in scikit-learn would be fine.

You can test each machine learning algorithm using cross-validation. Basically you split up your training data into a train and test data set. Run your algorithms on the train data set and see how they do on the test data sets. The accuracies of the algorithms on the cross-validation training set are approximations of the accuracies of the algorithms on unseen data.

• If you have time, can you please post some practical example as if I were 10 year old child. The linked wikipedia contains bunch of hard to understand math for me. Also I would like to ask if any ML method can be compared to any other (my 2nd question). I wold like to create some matrix or cheat sheet of ML. Thank you – Wakan Tanka Jun 8 '15 at 21:57
• @WakanTanka: Suppose I have data from the last week as to whether it rained or not. Suppose it is the following: YNYNYYN where the Y and N mean that yes it rained or no it did not rain. Now split this up like this: YNYN | YYN. The stuff after the | is the test data set. Train your models on the stuff before the |. – machineguyei Jun 8 '15 at 22:05
• Do you know about some practical materials (python or R would be fine) that could help beginner? – Wakan Tanka Jun 11 '15 at 8:56