0
$\begingroup$

I have a dataset with 19 features and one binary label (Error: Yes/No).

What I would like to find are the maximum values for each of the 19 features, where crossing them would mean having an unacceptably high risk of the "Error" value being a "Yes".

I'd do this, for example, by calculating the AUC of a feature based on a SVM or logistic regression algorithm and define the "optimal cutoff" of the feature as 0.5 of the AUC. And I'd do this repeatedly for each of the features.

Does this approach make good statistical sense?

$\endgroup$
0
$\begingroup$

No. Half of the area under a signal-detection curve need not correspond to whatever you believe unacceptably high risk is.

Presumably, by "risk" you mean "probability". In this case you want a probabilistic classifier such as a logistic-regression model (but not an SVM, at least without added machinery to produce a probability estimate) so you can estimate the probability of an error for each case. Then you can judge whether the probability is unacceptably high.

An alternative way to look at this task is in terms of ordinary classification rather than probabilistic classification, but with an asymmetric loss function. You can use any binary classifier for which you can choose the loss function. Choose the loss function to reflect how bad false alarms are compared to misses in declaring an error.

$\endgroup$
  • $\begingroup$ Thanks for the answer! I'm aware of the fact that 0.5 of the AUC doesn't necessarily make sense, depending on the subject matter your dataset is concerned with. The main point I was unsure about is whether the approach in terms of doing this evaluation with every single variable and the class label makes sense. What I got from your comment was that it would be a sensible structure of the analysis. Am I right about that? $\endgroup$ – Dionysos May 29 '17 at 17:27
  • $\begingroup$ @user32625 You can do that, but to do so would be to throw away any ability to find additive or interactive effects between the features. The ability to estimate such effects is why people use multiple regression instead of an ensemble of simple linear regression models. $\endgroup$ – Kodiologist May 29 '17 at 17:53
  • $\begingroup$ @user32625 If I answered your question to your satisfaction, you can accept my answer by clicking on the check mark under the voting arrows. $\endgroup$ – Kodiologist Jul 9 '17 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.