4
$\begingroup$

I'd like to classify a set of 3d images (MRI). There are 4 classes (i.e. grade of disease A, B, C, D) where the distinction between the 4 grades is not trivial, therefore the labels I have for the training data is not one class per image. It's a set of 4 probabilities, one per class, e.g.

0.7   0.1  0.05  0.15
0.35  0.2  0.45  0.0
...

... would basically mean that

  • The first image belongs to class A with a probability of 70%, class B with 10%, C with 5% and D with 15%
  • etc., I'm sure you get the idea.

I don't understand how to fit a model with these labels, because scikit-learn classifiers expect only 1 label per training data. Using just the class with the highest probability results in miserable results.

Can I train my model with scikit-learn multilabel classification (and how)?

Please note:

  • Feature extraction is not the problem.
  • Prediction is not the problem.
$\endgroup$
3
$\begingroup$

Let me try to answer this, I will edit the answer as I have more information. In general scikit-learn does not provide classifiers that handle the multi-label classification problem very well. That's why I started the scikit-multilearn's extension of scikit-learn and together with a lovely team of multi-label classification people around the world we are implementing more state of the art methods for MLC.

First of all, the question is do you need probabilities or just an estimate of how sure a classifier is. Not always the exact probabilities are what you can get very cheaply. I understand that you want to get probabilities P(A|X), P(B|X) etc. for a given instance X.

A. The simplest case: labels are indpendent i.e. P(A and B|X)=P(A|X)P(B|X). If such case occurs you can use scikit-multilearns Binary Relevance classifier's predict_proba:

here's a simple example with SVC as the per label probability estimator:

from skmultilearn.problem_transform import BinaryRelevance
from sklearn.svm import SVC

classifier = BinaryRelevance(classifier = SVC(probability=True),
    require_dense = [False, True])

classifier.fit(X_train, y_train)
probabilities = classifier.predict_proba(X_test)

This will estimate per label probabilities and then renormalize them. Unfortunately Binary Relevance may fail to detect a rise/fall of probabilities in case when a combination of labels is mutually or even totally dependent, it just happens.

B. If your labels are not independent you need to explore the data set and ask yourself what is the level of co-dependence in your data. There are several ways to handle dependencies. If you really expect a total dependence, a Label Powerset approach may be better, where each combination is treated as a separate class and probability will be estimated per that class. Note that this transformation is a hard one to perform, due to label imbalances and the underfitting nature of Label Powerset transformation, I've created a solution for this to divide the label space into interconnected subspaces - a data-driven approach to detect dependencies and split the problem into interally more dependent subproblems - see the data-driven approach to multi-label classification paper.

An example how to use it is here: http://scikit.ml/api/classify.html#ensemble-approaches - just use predict_proba instead of predict. Also you might want to change the clusterer to:

clusterer = IGraphLabelCooccurenceClusterer('fastgreedy', weighted=True, include_self_edges=False)

so that the label partition is more granular. It detects clusters of co-occurring labels and then calculates joint distributions P(A1,...,An|X) for labels A1...An per cluster, i.e. it expects the clusters of labels to independent.

If you like it and use this method please cite both the data-driven paper and the arxiv paper of scikit-multilearn, we can get funding to develop the library that way :) Also I'd love to know how the method worked for you so I can maybe improve it.

I find it easiest to just start with something, so if I were you, I'd go ahead and check the approach from point A and see what level of result you're getting. Then I'd try the label space partition approach. I need to write a tutorial on how to use it to explore the relations, will add this to my documentation todo list.

$\endgroup$
  • $\begingroup$ Thank you for these proposals and explanations. I will do as you suggest and tell you about the outcome. Unfortunately, I don't have enough points to vote for your answer (yet). $\endgroup$ – texasWINthem Oct 31 '17 at 7:04
  • $\begingroup$ Thank you! My labels are not independent and therefore I replaced the classes by combinations of classes. As a proof of concept I did the pre-processing manually - with quite good results. I will now automate it and find the optimal parameters, classifiers, etc. As soon as I have the needed reputation, I will come back and vote for this answer. $\endgroup$ – texasWINthem Nov 2 '17 at 6:56
  • $\begingroup$ You can use LabelPowerset classsifier for the simplest case if you don't have large numbers of labels. $\endgroup$ – niedakh Nov 3 '17 at 9:02
  • $\begingroup$ @niedakh Did you address the requirement to model probabilistic labels? ie it's not just 1-hot encoded multi-label, it's a probability distribution across the labels... $\endgroup$ – eggie5 Oct 25 '19 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.