2
$\begingroup$

I am starting my adventure with machine learning and there is one concept I don't understand, so:

How one does determine classifier accuracy in real life machine learning system?

Let me give you an example I have in mind - lets say I have classifier ensemble consisting of 3 classifiers, data come in chunks. The pruning criterion is somehow correlated to the accuracy of each classifier in the ensemble (for the sake of this example lets just say that in every iteration the "weakest" model is dropped and replaced by another classifier from the pool). In "laboratory environment" I can easily check the accuracy of each classifier, because I know the class of the every object in the data set (assuming it is supervised learning) - but in the real life system I don't have that kind of information. The only class I have for given object is the one predicted by the classifier, so how do I determine how "strong" each model is so I can drop the weakest one?

This question might be trivial, but I would appreciate some explanation on this.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I could help you better if you roughly describe the application scenario.

In the medium to long run, you can often find out what the true class was. For example, when you classify clients into credit worthy and not worthy, you will eventually find out how many of those that you have given credits to will default. The question is whether such feedback would arrive soon enough to be actionable in model selection.

Even if you don't get feedback on a data-point per data-point basis, you might have more indirect feedback that you can use for model selection. For example, you could compare models based on the average revenue or profits per customer that were achieved using the models to classify the customers in a business context. Or you could proxy classification performance through the prevalence of customer complaints and help-desk tickets (fewer would be better here) if there is no profit notion involved.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am projecting system for predicting Trust Value for my engineering thesis - this system does have implemented classifier ensemble that uses pruning criterion that is combination of each model accuracy and ensemble diversity. Basically it predicts if two users of a social network would "hit it off", so it could, for example, "propose them to be friends" - like in some social websites it displays "users you might know". Now it all works fine, but I couldn't imagine it being implemented in real life just because of this "model accuracy factor" in pruning criterion. $\endgroup$
    – Johnny Johansson
    Commented Nov 14, 2016 at 18:24
  • $\begingroup$ But in this case the real life feedback is quick and easy. Define "hit it off" as accepting the suggested friendship in lets say 2 days or less and define "not hit it off" as taking longer or never accept the suggested friendship. The classifier predicts if they hit it off and maximum two days later you know if it was right or not. If this delay is already too long, you should perhaps not use that combination of algorithms that requires such immediate feedback in this application context. $\endgroup$
    – David Ernst
    Commented Nov 14, 2016 at 18:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.