I am training SVM by a dataset with 8 features using 10-fold CV.
The AUC for testing data is under 0.5. I remember that somewhere it had been written that in cases with AUC < 0.5, we can inverse the answer of the classifier to test samples. For example, in a two class problem, if SVM predicts class '1' for a test sample, we consider it as class '2'.
Is it correct?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ How much below .5? If it is very far below .5, I would speculate there is a code error. If it is .493, this might just indicate chance performance because there is no pattern detectable by an SVM of the kind you're applying in the data. $\endgroup$– jonaCommented Feb 15, 2016 at 14:33
-
$\begingroup$ It is about 0.4-0.46. Indeed, I do not expect to get high AUC with these features, but I do not expect to get AUC below 0.5 too. $\endgroup$– user4704857Commented Feb 15, 2016 at 14:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes- mathematically, in a two-class problem where the AUC is < .05, the prediction of the model can be reversed to get a better AUC fit on any (one) set of data.
That said- if you're getting an AUC this low (worse than chance) on the test set, the model is more-or-less telling you that it cannot (1) determine a useful relationship between the variables and/or (2) that the relationships the model is using are not generalizable outside of the training set- i.e. the model is over-fitting to training noise.
Since you seem to know in this case that you have weak features- it indicates to me that the model may be fitting only to training noise. That is, your features are not informative of the class separation and only random variation is being modeled. I can't say this for certain without contextual information, but it would be one reasonable hypothesis to be tested in this situation.
