Let's assume there is a function of interest:
y = abc
...where a, b, and c are the probabilities of independent events. I've created classification algorithms (specifically, SVMs) to estimate those three probabilities (with isotonic regression performed on the raw probability outputs for scaling purposes).
My concern is that multiplying the probabilities may still be invalid due to (1) unbalanced classes for all three classifiers, and (2) different proportions of imbalance between the classifiers:
There are 4000 instances in which clf_a = 0.
There are 1000 instances in which clf_a = 1.
There are 3500 instances in which clf_b = 0.
There are 1500 instances in which clf_b = 1.
There are 1500 instances in which clf_c = 0.
There are 3500 instances in which clf_c = 1.
Because classifier_c has only 1500 instances supporting class 0 while the other two classifiers have support of 3500-4000 each (and vice versa for class 1), the resulting accuracies (and other summary statistics) for the classifiers are rather different. For example:
clf_a recall for 0: 0.96
clf_a recall for 1: 0.84
clf_b recall for 0: 0.95
clf_b recall for 1: 0.87
clf_c recall for 0: 0.81
clf_c recall for 1: 0.98
Is my concern about the differing proportions of support for the three classifiers a valid one, even though I'm scaling the initial probability outputs from the classifiers (again, via isotonic regression, although I see similar results when using Platt scaling)? If so, are there steps I can take to resolve the issue, such that I can eventually multiply a, b, and c? Or are there more basic flaws with this approach?
I cannot acquire additional data. I have yet to try oversampling and undersampling. I've tried applying a cost matrix for class weights to the SVMs, though, and summary statistics are largely unchanged. I have also tried other classifiers; the situation is the same for all of them, so this isn't an issue specific to SVMs.
I should emphasize that while I want to see robust, powerful classifiers, my primary goal is to estimate probabilities for which subsequent multiplication is mathematically valid.
Thank you in advance for any thoughts, comments, or ideas you might have.