3
$\begingroup$

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would compare the usefulness of each combination.

For example I can both train an SVM using Features 1, 10, 15 to get 100% accuracy. But at the same time I can train a logistic regression classifier only using Feature 7 to get 100% accuracy. Also this is a binary classification problem.

$\endgroup$
  • 1
    $\begingroup$ Could you please elaborate on the settings of the cross-validation procedure (i.e., number of folds). One way to put a classifier to the test is use small number of folds. Another one is to add some noise to the test data in order to test its robustness. Also check your data, scoring 100% all times means that your problem is most probably binary separable. $\endgroup$ – Dimitrios Bouzas Feb 15 '14 at 16:17
  • 1
    $\begingroup$ -1 Please don't cross-post. meta.stackexchange.com/questions/64068/… $\endgroup$ – Sycorax says Reinstate Monica May 11 '16 at 12:48
5
$\begingroup$

The sample size of your test data can be estimated using probability inequalities. You can calculate the required sample number to get a desired accuracy with high probability.

Yet with small test sample sizes, even if a decent classifier can be derived, it cannot be shown that the classifier works well. You may want to have a look on the paper discussing this problem: Beleites, C. et al.: Sample size planning for classification models., Anal Chim Acta, 760, 25-33 (2013). In the paper, with a total of 34 cases you won't even be able to get a useful estimate of the learning curve because of the small test sample size (≤34). The curve in the paper might be helpful to you.

$\endgroup$
3
$\begingroup$

The only reasonable solution is to gather more data. If some models are perfect then they are perfect, you cannot compare them. Obviously you can analyze which is simplier (has less parameters), build simplier model (in terms of VC dimension) or learns faster, but the fact is - if your data is so simple, that you get 100% accuracy, there is nothing really to analyze (unless these scores are result of incorrect evaluation procedure, which may also be the case).

$\endgroup$
  • $\begingroup$ Hmm, not sure if I am in the position to collect more data. Would AUC from the ROC curve be a measure of "robustness" of the resulting classifier output? $\endgroup$ – user2422566 Feb 10 '14 at 22:06
3
$\begingroup$

There's value in simplicity. Let's assume your logistic regression model trained on feature 7 had intercept $a$ and coefficient $b$ for feature 7. This means the model predicts 1 whenever feature 7 exceeds value $-a/b$ and predicts 0 otherwise. You get the best of both worlds using this model -- your model is extremely simple to describe and implement, and at the same time it has perfect cross-validation accuracy.

$\endgroup$
3
$\begingroup$

You want to use a proper scoring rule here: Brier Score, Logarithmic Loss or Spherical Loss. The idea is that the best model is the one that better predicts probabilities, so you can actually use the outputted probabilities to make decisions.

$\endgroup$
2
$\begingroup$

ROC / AUC does not help here. If the binary classifier is perfect, and it has some threshold to vary (or else ROC does not make sense) then it scores all positive examples at or above the threshold, and negative examples below. As the threshold goes from low to high, the ROC curve goes from (1,1) to (0,1) to (0,0). The AUC is always 1.

You can discriminate amongst perfect classifiers, but it's going to take some more information. If you don't have more data, but your classifier is one that can compute a confidence or probability for a classification, then you want the one that is most confident about its answers. For example you could take the one with lowest differential entropy over the confidences in the positive / negative examples.

$\endgroup$
  • 2
    $\begingroup$ This is basically correct; the concept that you're dancing around is proper scoring rules like log-likelihood or Brier score $\endgroup$ – Sycorax says Reinstate Monica May 11 '16 at 12:56
  • $\begingroup$ @Sycorax: Are you saying there is some relation between proper scoring rules and (differential) entropy? Can you elaborate, cite ... ? (I can ask this as a question if you want to answer) $\endgroup$ – kjetil b halvorsen Dec 11 '18 at 13:09
  • $\begingroup$ @kjetilbhalvorsen I think that the relation is indirect at best, since I'm not aware of a differential entropy interpretation to the Brier score, yet the Brier score is a proper scoring rule. This would make a good question though, since I'd like to know a more definitive answer! $\endgroup$ – Sycorax says Reinstate Monica Dec 11 '18 at 15:33
1
$\begingroup$

Comparing such models using their performance metrics does give you additional information, as mentioned with the other answers already. But in terms of e.g. models choosing, the types of models scoring so well give you additional information. The less complex the model that can solve your problem sufficiently (in your case: perfectly), the better. I would consider logistic regression to be very simple in this regard. This means: using a logistic regression model will be better than using a more e.g. an ANN or RF, and better than a SVM (except for the linear kernel, which boils down to also using a linear separation hyperplane with just a more complex fitting process, but a similar resulting separation complexity).

Essentially, this also applies to features: using less features is better (be aware that this strongly depends on how well your model generalizes: you will almost always find a way to obtain perfect results with a handful of features when dealing with very few samples, even when using simple models, but the resulting model will not generalize well).

If you achieve such good results with such simple models, this also tells you something about your problem and data. If you have very few samples, you should consider obtaining more, because your models probably don't generalize well yet. But if you are using a sufficient amount of samples already, this probably means your problem is easy to solve using only few features.

This is where you might want to thoroughly visualize the features and features-target-variable-relations to get an in-depth understanding of their relations, and where you would want to cross-check those relations with your domain specific knowledge about the problem, to ensure that those relations actually make sense. If not, it could e.g. mean that something is wrong with your data in the first place.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.