I am not expert in this area so please bare with me. Is it possible to somehow evaluate the success rate of machine learning algorithm/methods. I suppose it could be done this way: Give a various ML one dataset and then check which one achieves the best score. This assumes that the correct outcome is already know and it will be used as a reference during comparison with ML. The problems that comes to my mind are:
- various ML need to have various input formats, but this could be solved using some text preprocessing etc. (I do not know this for sure I'm just thinking aloud)
- some ML are primary predetermined for specific tasks, so it would be best to compare "similar" families of ML algorithms?
Is there any study or even better the whole framework for this purpose? I am starting with ML and would like to try several algorithms and compare their results performance etc. Something practical in scikit-learn would be fine.