Machine Learning - How to use a classifier to find the most likely model

I have been learning about the use of machine learning algorithms and their application to particle physics.

Now, I have some doubts concerning what to do with the results. Let me explain: imagine that we have two theoretical models to explain the data. I go through all the process of setting up the discriminating variables, choose the method, train the classifier and test it against overfitting, the works.

Consider then that at the end of the process I have a classifier that has an area under the ROC curve of, say, $0.7$. If I understand it correctly, that means that from a set of $N$ events, $70\%$ will be correctly classified (is this right?)

My question is then, what to do with this classifier, or as a matter of fact, how can we use a statistical model, with some non null probability of misclassification in order to infer which theoretical model is the one that most likely explains the experimental data?

The question seems like something that should be in any statistical book but I can't seem to find any reference that explains this issue.

If I understand your situation correctly, you've misapplied the idea of classification. A classifier is a procedure for taking an observation which belongs to one of several classes, each of which is assumed to exist, and predicting its class. For example, a classifier might take the length of a person's name (an integer) and predict the person's gender (male or female). This isn't your situation because you don't have observations each of which could have been produced by one of two processes, with both processes definitely existing; rather, either all your observations are produced by one process, or all of them are produced by the other process. The statistical tool you want for this is model selection, which is indeed a core topic in statistics you can read about in most statistics books.

Consider then that at the end of the process I have a classifier that has an area under the ROC curve of, say, 0.7. If I understand it correctly, that means that from a set of events, 70% will be correctly classified (is this right?)

No, those are different animals. Proportion classified correctly is applied to a single classifier and is computed by simply counting how many of the observations in the test data are correctly predicted. ROC curves can be created for a model which can produce a classifier for any setting of a classification threshold (such as a logistic-regression model), and the area under ROC curves is difficult to interpret, which is one reason that computing and reporting the area under an ROC curve is of dubious value.

• Thank you for you answer and providing some names for me to browser. I think I understood all of what you wrote but my question remains.So, I have two theoretical models. I set up some simulated events considering one or the other model, which I shall then use to train some classifier such that it "learns" to separate events from one model or the other. Now I agree with you, real experimental data will either be better - more likely - explained by, hopefully, one model or the other. So, how is that classifier used to determine the most likely model? – PML May 29 '16 at 21:51
• @PML I'm saying that doesn't make any sense. You don't need a classifier. Just use a standard model-selection technique, like computing the BIC or comparing cross-validated prediction error, to choose between your two models. – Kodiologist May 29 '16 at 22:46