I have a multiclass classification problem that I have solved using a 'one vs. rest' approach via binary logistic regression classifiers from Python's scikit-learn package. In my problem, there are 3 classes for which I am trying to predict probabilities. I have trained 3 logistic regression classifiers (i.e. 1 for each of the classes), but how should one combine the probabilities from the independent classifiers?
Based on reading through the literature, there is no reason for the probabilities to add up to 1 when considering binary logistic regression for a multiclass problem. Yet when I use my trained logistic regression classifiers, the summed probabilities from all 3 of them always add up to exactly 1.
After further investigation, I have discovered the following: I can extract the coefficients and obtain the following parameters (i.e. 4 weights for the 4 input features and bias terms) for my 3 logistic regression classifiers:
weights = array([[ 0.11948853, -3.2523997 , 1.81306023, -0.29211884], [ 1.16800984, 2.32887278, -1.72453382, -1.17726167], [-2.39495012, 1.6836802 , -0.83319772, 1.86957419]]) intercepts = array([ 2.79030646, -3.89132394, -4.53314668])
I computed the probabilities myself by plugging an example set of input features (which correspond to different inputs at different times) into a sigmoid, and arrived at the following predicted probabilities from my classifiers:
Clearly, the probabilities do not add up to 1. Yet, when I instead use the
predict_proba routine from scikit-learn, I arrive at the following input:
The probabilities now do add up to 1 as I noted above. It appears some normalization has been applied, but does anyone know what the normalization procedure applied here is by scikit-learn? I've tried attempting multiple approaches to normalize the probabilities (e.g. normalizing the individual probabilities by the squared sum, applying softmax to either the probability or linear predictor), and although they're close, they do not quite exactly match the above output from
predict_proba. Any ideas on what exactly scikit-learn is doing here to combine the probabilities?