Skip to main content
32 votes

Multilabel classification metrics on scikit

The subset accuracy is indeed a harsh metric. To get a sense of how good or bad 0.29 is, some idea: look at how many labels you have an average for each sample look at the inter-annotator agreement, ...
Franck Dernoncourt's user avatar
28 votes

How to compute precision/recall for multiclass-multilabel classification?

For multi-label classification you have two ways to go First consider the following. $n$ is the number of examples. $Y_i$ is the ground truth label assignment of the $i^{th}$ example.. $x_i$ is the $...
phoxis's user avatar
  • 441
22 votes

How do you calculate precision and recall for multiclass classification using confusion matrix?

Using sklearn or tensorflow and numpy: ...
Cristian Garcia's user avatar
15 votes

How to build a confusion matrix for a multiclass classifier?

While there are some answers already on this forum I thought I'd give the explicit equations to make it more definite: Assuming you have a multi-class confusion matrix of the form, \begin{align} C=\...
Josh Albert's user avatar
15 votes
Accepted

Multi-label or multi-class...or both?

Definitions. In a classification task, your goal is to learn a mapping $h: X\rightarrow Y$ (with your favourite ML algorithm, e.g CNNs). We make two common distinctions: Binary vs multiclass: In ...
galoosh33's user avatar
  • 2,302
14 votes
Accepted

Many binary classifiers vs. single multiclass classifier

Your Option 1 may not be the best way to go; if you want to have multiple binary classifiers try a strategy called One-vs-All. In One-vs-All you essentially have an expert binary classifier that is ...
Pablo Rivas's user avatar
13 votes
Accepted

Is GINI limited to binary classifiers or can we use it for multi-class classifiers as well?

The Gini impurity can definitely be used to quantify variance in a multi-class setting, not only in the binary case. Gini impurity is defined as $$ G(p) = \sum_{i=1}^{J}{p_i} \sum_{k \neq i}^{J}{p_k}...
Simon's user avatar
  • 298
12 votes
Accepted

How the probability threshold of a classifier can be adjusted in case of multiple classes?

You can use a prior distribution over the classes. Let us assume that your model computes a vector of class probabilities $v$. You can define a vector of prior probabilities $\pi$ and then compute ...
cangrejo's user avatar
  • 2,251
11 votes
Accepted

Using regression where the ultimate goal is classification

Welcome to the site. The first part of the first choice seems much better to me; you could use some form of count regression, probably negative binomial (the assumptions of Poisson regression are ...
Peter Flom's user avatar
  • 122k
10 votes
Accepted

How to build ROC curve (or AUC) of classification model from confusion matrix only

You cannot construct a ROC curve from the confusion matrix alone. A confusion matrix represents a single point in the ROC space, and you need all possible confusion matrices at all thresholds to build ...
Calimo's user avatar
  • 3,674
9 votes

Neural network for multi label classification with large number of classes outputs only zero

Tensorflow has a loss function weighted_cross_entropy_with_logits, which can be used to give more weight to the 1's. So it should be applicable to a sparse multi-...
tobigue's user avatar
  • 273
9 votes

How to compute accuracy for multi class classification problem and how is accuracy equal to weighted precision?

I've got a wonderful solution and a perfect understandable solution for this problem as I was looking for same from this Question You can calculate and store accuracy with: ...
Ananda G's user avatar
  • 201
9 votes
Accepted

Poor multiclass classification using Caret in R

I think that the problem is not that you are using the classification methods poorly, but rather that this data has little predictive power for the regions. First of all, two classes have little ...
G5W's user avatar
  • 2,630
8 votes

What is the difference between Multitask and Multiclass learning

Just to give a more clear understanding, I have explained each terminology with examples Multiclass classification/(One-Vs-One and One-Vs-ALL): Classification task with >2 classes! ...
Anu's user avatar
  • 673
8 votes

How to threshold multiclass probability prediction to get confusion matrix?

According to @cangrejo's answer: https://stats.stackexchange.com/a/310956/194535, suppose the original output probability of your model is the vector $v$, and then you can define the prior ...
allenyllee's user avatar
8 votes
Accepted

How to apply Softmax as Activation function in multi-layer Perceptron in scikit-learn?

I suposse that the Softmax function is applied when you request a probability prediction by calling the method mlp.predict_proba(X). To support my supposition I ...
Daniel López's user avatar
7 votes

How to apply Softmax as Activation function in multi-layer Perceptron in scikit-learn?

The MLPClassifier can be used for "multiclass classification", "binary classification" and "multilabel classification". So the output layer is decided based on type of Y : Multiclass: The outmost ...
Trideep Rath's user avatar
7 votes

Better performance using Random Forest one-Vs-All than Random Forest multiclass?

I had exactly the same question as you, and was a bit sad to find out no answers were posted on your topic... That said, I found this paper : One-Vs-All Binarization Technique in the Context of ...
clement g's user avatar
  • 171
7 votes
Accepted

Imbalanced multiclass classification with many classes

There is no real answer to your question, because it really depends on what you are trying to archive, i.e. is your goal to get a very high classification accuracy or is it rather data exploration? ...
Scholar's user avatar
  • 1,045
7 votes
Accepted

Error while performing multiclass classification using Gridsearch CV

Accuracy might look tempting but not a good metric in general. In multilabel classification, for each class we'll have f1 score, ...
gunes's user avatar
  • 57.5k
6 votes
Accepted

Accuracy vs Jaccard for multiclass problem

The issue has been reported on scikit-learn GitHub repository: multiclass jaccard_similarity_score should not be equal to accuracy_score #7332 scikit-learn's Jaccard score for the multiclass ...
Franck Dernoncourt's user avatar
6 votes
Accepted

Optimize classification rule in multinomial logistic regression

The question is ill-posed. If you are trying to optimise some function of specificity and sensitivity (other than accuracy) for a logistic regression model by altering the classification threshold, ...
Dikran Marsupial's user avatar
5 votes

Maximum number of classes for RandomForest multiclass estimation

I have at least one experience doing so. For the NHTS 2017 dataset, I have modeled a number of variables. Notably, random forests perform quite well on predicting vehicle ownership per household (...
Him's user avatar
  • 2,257
5 votes
Accepted

Training N classifiers for N labels vs one classifier with N labels

It mostly depends on your labels. Training one classifier with N labels (= multitask) makes more sense when labels are related. But one cannot predict in advance with 100% certainty whether multitask ...
Franck Dernoncourt's user avatar
5 votes

Calculate accuracy using true/false positives/negatives

You are confused about the terminology. The terms "false positive" and "false negative" are only used in binary classification. You have 3 classes, so, these terms aren't applicable. However, we ...
Haitao Du's user avatar
  • 37k
5 votes
Accepted

Multi-class classification

Suppose a point lies in the green region, and look at the classifier pairs starting in the lower left, moving counter-clockwise. The first classifier pair says 'prefer c2 over c1' the second ...
Max S.'s user avatar
  • 1,746
5 votes
Accepted

Handle categorical class labels for scikit-learn MLPClassifier

Using MLPClassifier you can do exactly what you suggested, that is represent classes as integers from 0 to 27 (in the case of 28 classes). Here is an example with MLPClassifier and MNIST dataset. You ...
piotrwiercinski's user avatar
5 votes
Accepted

Does watermark/text on images at the same position influence the classification of images using CNN?

Yes, it can be a problem. A very similar example was used in the Unmasking Clever Hans Predictors and Assessing What Machines Really Learn paper by Lapuschkin et al (see below). They show an example ...
Tim's user avatar
  • 139k
5 votes
Accepted

How to compare labels from clustering analysis and original ones?

A common approach is to consider all pairs of points and count for each pair whether the two points are assigned to the same cluster or to different clusters. This yields four numbers: a = #SS = ...
cdalitz's user avatar
  • 5,192
5 votes

Using regression where the ultimate goal is classification

“High risk” of what? You are predicting the number of failures, so if you aim to predict if there's a risk of failure, anything close to or higher than 1 failure is a risk. On another hand, if you ...
Tim's user avatar
  • 139k

Only top scored, non community-wiki answers of a minimum length are eligible