0
$\begingroup$

I am starting with Machine Learning and I have some trouble identifying true negatives and false positives from my classified data set. I have a classifier which classifies items in three classes.

Now suppose from my classification results I have the following:

  • 5000 total result items
  • 4000 from them can be classified (they have some feature with data which is useful for the classifier) and the remaining 1000 could not be classified (they have no useful "field" for my classifier)
  • 1000 (from the 4000) were the hits for all three classes.
  • 0 of those 1000 hits were false positives

Now the true negatives (which are those correctly rejected) are counted from the total result items (5000) or from those which have data suitable for classification (4000)?

I am a bit confused by the fact that other question asked about TP, FN, etc. taking each true, false, positive, negative for each of their classes instead for the whole 3 classes as I did for my data set. What I have missed?

Improve this question
$\endgroup$
3
  • $\begingroup$ What does the reference say? Do all the 5000 items have a reference class? Also, what is the status of your 3000 (from the 4000) that are not "the hits for all three classes"? Anyway, if the initial 5000 items are representative of your real data, then all of them should be used when computing precision and recall. Which mean that if you only provide a class for 1000 items, your recall cannot be more than 20%. $\endgroup$
    – jpl
    Commented Jan 5, 2015 at 7:27
  • $\begingroup$ What do you mean with reference class? The remaining 3000 (from the 4000) belong to other classes which I am not interested in my case study. $\endgroup$
    – Veronica
    Commented Jan 5, 2015 at 17:18
  • $\begingroup$ OK, but even if you're not interesting in some classes, you cannot just throw the corresponding instances away, without even looking at what you predicted. Then, TPs are the number of correctly classified instances of your three 'interesting' classes over the 5000 instances, and all the same for TNs, FPs and TNs. $\endgroup$
    – jpl
    Commented Jan 6, 2015 at 7:30

2 Answers 2

Reset to default
1
$\begingroup$

We aim at evaluating the performance that the model would have on 'real-life' instances. So, if your initial 5000 instances are representative of real data, then you have to consider true positives, true negatives, false negatives and false positives over the 5000 instances, regardless of whether you are 'interested' in their classes or not.

However, probably, overall precision and recall is not your main interest here, since you are only interested in three classes. Then, you should compute precision and recall for each of these three classes separately.

Also, you should probably train your classifier to classify 4 classes: A, B, C and OTHER, which would be an aggregation of all non-interesting classes.

Or, in a hierarchical way, if the class types fit better to this way of doing: a first classifier is trained with 2 classes (INTERESTING and NOT_INTERESTING), and then a second classifier consider only interesting classes and separate them into A, B and C.

Improve this answer
$\endgroup$
0
$\begingroup$

To understand how successful/accurate the classifier training/learning was, do all your metrics on the data set which was actually presented and used by the classifier - 4000 examples in your case as I understand.

Precision and Recall typically are calculated for each class independently.

Also, you can combine different classes into one or split them in multiple based on your real-world interpretations and needs.

What you loose by focusing on just 4000 is the overall utility of the classifier on real world data. The classifier might work well on 4000/usable datasets but the fact is that it can handle only 80% of data the rest won't be usable by the classifier. This may or may not be of significance.

Example:

A doctor can help with general diseases(physician), lung related diseases and heart related diseases but is not a specialist in orthopedics(bone related troubles).

In this case patients with bone related problems will be turned down by the doctor(ideally) while to others the doctor will prescribe something. This is like 1000 getting rejected out of 5000.

Now out of the 4000 some might get good treatment while some bad which will tell you how good the doctor is in each class of disease for which he handles the patients (physician, lung problems, heart problems) if precision, recall and other metrics are calculated independently for each class on 4000

On 5000 if you calculate metrics like precision you will first have to create a class called turned down by doctor. And then find precision and recall for this (and others too) which should be 1 if doctor is able to identify and turn down orthopedic and diseases beyond his ken accurately. (in your world your classifier is able to turn down the data records accurately as you drop anything and only those things which don't have any of the feature/field needed). This precision and recall would represent how good the doctor is with all the patients where definition of being good is being good at :

  1. correctly prescribing to patients with known disease
  2. correctly turning down patients who have disease beyond his ken/expertise

Hope this helps

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.