0
$\begingroup$

We're using a decision tree based classifier that assigns record to one of two classes (true /false). Model is producing answers with probability of this value. So to check how good are answers the squared error is used 

(expected{0, 1} - result{0.0 .. 1.0})^2

This method gives quite good picture of learning curve, but the business problem is defined as find records that can be classified as "true" with highest probability. So from the business point of view even drastically biased results are ok, as long as records that are expected to be "true" gets even slightly higher ranks that those that are expected to be false.

My question is:

How to define metric to measure quality of this model from business point of view?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

The business value may relate to some monetary value of wrong/bad decisions. You can introduce the prices of:

  • false positive
  • false negative
  • true positive
  • true negative

Then you can quantify the expected/average value of the classifier for any threshold. As a result, you can get the optimal threshold.

From the business perspective, it makes sense to evaluate also the quality of the existing decision-making mechanism and compare it with your classifier.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.