improving classification accuracy of the dataset as a whole by considering classifier distributions

Question

Overview

I'm new to machine learning so apologies if I misuse terms. I have an idea to improve my classification analysis that I feel is not terribly unique, but I can not find a reference to such a procedure with my limited knowledge. In short, I am using ML classification to try to partition my data. That is, I don't care for any given sample what class it is, but I want to know the fraction of each class in my dataset.

Example problem

Consider a binary classification, where I have some signal $S$ and background $B$. I train up my classifier, and get the following confusion matrix:

+---+-----+-----+
|   |  S  |  B  |
+---+-----+-----+
| S | 0.9 | 0.1 |
| B | 0.2 | 0.8 |
+---+-----+-----+

That is, 90% of signal is correctly identified and 80% of background.

Now, I give my classifier a set of testing data with a 20/80 S/B split, and this classifier tells me that my data consists of

$$ \begin{pmatrix}S_0 \\ B_0\end{pmatrix} = \begin{pmatrix} 0.9 & 0.2 \\ 0.1 & 0.8 \end{pmatrix} \begin{pmatrix} S \\ B \end{pmatrix} = \begin{pmatrix} 0.34 \\ 0.66 \end{pmatrix} $$ Not super great.

Improvement #1

I think I am allowed to use my knowledge of the category leakage (expressed in the confusion matrix) in order to correct my population estimates. I.e.,

$$ \begin{pmatrix}S_1 \\ B_1\end{pmatrix} = \begin{pmatrix} 0.9 & 0.2 \\ 0.1 & 0.8 \end{pmatrix}^{-1} \begin{pmatrix}S_0 \\ B_0\end{pmatrix} $$ which in this case gives back the input signal and background perfectly. In a real case of course the mismatch won't be exactly the same as the confusion matrix on the training data, but it must still be better than taking the numbers at face value, right?

Improvement #2

Let's say I modify my classifier so that, for a given input sample, if the classification probability is below some threshold (e.g. 10%), classify as "Uknown". Now I have a new confusion matrix:

+---+------+------+------+
|   |  S   |  B   |  U   |
+---+------+------+------+
| S | 0.9  | 0.02 | 0.08 |
| B | 0.05 | 0.75 | 0.2  |
+---+------+------+------+

and when I give it the same 20/80 split, I get $$ \begin{pmatrix}S_0 \\ B_0 \\ U_0 \end{pmatrix} = \begin{pmatrix}0.22 \\ 0.604 \\ 0.176\end{pmatrix} $$

Now to estimate the true signal and background fractions, I could perform e.g. a $\chi^2$ minimization of the different weights in the confusion matrix to my test output. I.e. minimize $|\mathbf{\epsilon}|^2$ in

$$ \begin{pmatrix}S_0 \\ B_0 \\ U_0 \end{pmatrix} = \mathbf{M_c} \begin{pmatrix} S_1 \\ B_1 \end{pmatrix} + \vec{\mathbf{\epsilon}} $$ where $\mathbf{M_c}$ is the confusion matrix (transposed).

Improvement #3

For any sample I give to the classifier, I get a score that tells me the probability that the sample is signal. Using my training data (or better, a different, independent set of training data), I can build PDFs for the score distribution separately for signal and background events. When I want to evaluate a test dataset, I get the distribution of scores from the classifier and fit that to the weighted sum of my signal and background PDFs, and the resulting weights give me the fraction of signal and background in the test dataset.

Conclusion

Is this sort of approach standard, and where can I find more information?
If it's not already standard, are these approaches valid, and likely to add anything useful in a real-world scenario?

Answer 1

If the problem is to estimate the relative class frequencies in a test set (or operational conditions) that may differ from those in the training set, then the best approach is to use a probabilistic classifier that estimates the probability of class membership, and then apply the algorithm described by Saerens et al., which solves exactly that problem.

Saerens, M., Latinne, P., & Decaestecker, C. (2002). Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. Neural computation, 14(1), 21-41. (doi:10.1162/089976602753284446, pdf)

I've used this on a couple of projects and it has worked well.

Answer 2

I can't say I know the standard procedure for your field but I think the standard in statistics more broadly would be to use the probabalistic framework of your model, which I imagine is cross-entropy. That is $$P(Y|X)=E[Y|X]=f(X)$$ and this follows from the fact that $Y$ is binary. Note the assumption is that $f$ models the expectation exactly. Therefore, if we want to know that marginal expectation $$E[Y]=\int_X f(X)P(X) dX \approx \frac{1}{N}\sum_{x\in X} f(x)$$ You might also want to quantify the uncertainty in this point estimate. One way to do this is to model $f$ as a random function i.e. treat the parameters of your network as random variables as in Bayesian analysis. Then one could look at the distribution of expectations.

Finally, if you suspect that your training set under-represents the data space, you might want to model $P(X)$ and calculate all expectations using samples from the estimated density function instead.

As a general note, accurate point estimates on (a limited number of) test datasets with known porportions isn't necessarily the best way to evaluate success. Since this is a classic inverse inference type problem, finding a way to express certainty/uncertainty about your prediction on a given dataset is of much greater practical value. For example, you could treat the confusion matrix as a random variable produced by variation in the function $f$. Even though any particular estimator might yield values closer to a particular ground truth, in the absence of this knowledge, estimator uncertainty is much more informative in deciding whether or not to trust a value.

Answer 3

Did your trainingset have a 34%/66% S/B split by any chance?

Your model will have learned that, without taking any of the features of your samples into account, it is going to be right 66% of the time when predicting that a sample belongs to B.

What you can do is vary the threshold at which your model decides to which class a sample belongs. An ROC-curve can then be used to compare your models at different thresholds.