I'm evaluating a classifier to detect objects.

For example:
I have a set1 with 50 pictures with the object to detect.
And set2 with 100 pictures without the object.

I determine the true positves (0 in set2), true negatives (0 in set1), false negatives and false positives for those sets.

If I want to merge set1 with set2 (to have a set with and without objects and real values for TP and TP) I have to decrease the size of set2.


Can I do this with a simple division? (divide the values of TN, TP etc by 2) or do I have to randomly pick 50 pictures of set2 and use these TN's etc.?

Edit: Example:

Set1: TN = 43 FP = 6 FN = 1 TP = 0
Set2: TN = 0 FP = 13 FN = 0 TP = 87
Set2 divided by 2: TN = 0 FP = 7 FN = 0 TP = 44

Set12: TN = 43 FP = 13 FN = 1 TP = 44

I need an explanation to back this up. thx..


For the imbalanced data sets classification, downsampling is mostly used in the case when there is huge difference between the size of two groups, while for your case, maybe upsampling (aka duplicate your set1 to alter the balance) is a better strategy. You can also design your own strategy of resampling (bootstrap for example). Besides, studies have shown that Naive Bayesian classifier and decision tree algorithms are insensitive to the class rebalancing, so the performance of your sample merging may also be dependent on your sampling method. You can try several strategies to determine which one works best among them.

(You can also get more details on this topic from this pdf on imbalanced learning.)

  • $\begingroup$ I think upsampling is not an option (can't take new pictures and won't use them twice). Down- or undersampling: Seems one way is to choose 50 pictures by random (as mentioned in your linked paper) $\endgroup$ – ddd Nov 21 '13 at 11:51
  • $\begingroup$ I've updated the question with an example. $\endgroup$ – ddd Nov 21 '13 at 12:19
  • $\begingroup$ Why down or upsampling is required? Isn't it enough to use the correct prior on the classes? $\endgroup$ – Cagdas Ozgenc Nov 21 '13 at 12:32
  • $\begingroup$ not sure with the prior.. $\endgroup$ – ddd Nov 22 '13 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.