I'm currently analyzing the same data using lasso, elasticnet and random forest classifiers. I'm just wondering if there is a best way to compare these three models(eg. AUROC, accuracy, mcnemar's test), or I should compare multiple features and trying to figure out if they all favour the same model?
In machine learning, there is no one algorithm that’s always better than others which is as per the “No free lunch theorem”. Therefore, one has to try with different set of classifiers and choose the best one.
Now it is also important to statistically compare the classifiers to know whichone is the best one. One way is to carry out one-way ANOVA followed by post hoc tests using Tukey-Kramer method for multiple comparison analysis to determine which pairs of means are significant and which are not. This generates a graph, similar to box plot where two means are significantly different if their intervals are disjoint, and are not significantly different if their intervals overlap.
For more info, refer to this research paper
More useful links: