76

In a 2-hypothesis case, the confusion matrix is usually: Declare H1 Declare H0 Is H1 TP FN Is H0 FP TN where I've used something similar to your notation: TP = true positive (declare H1 when, in truth, H1), FN = false negative (declare H0 when, in truth, H1), FP = false positive TN = true negative From the raw data, the values in the table would ...


52

Multiclass classification means a classification task with more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Multiclass classification makes the assumption that each sample is assigned to one and only one label: a fruit can be either an apple or a pear but not both at the same time. Multilabel ...


32

Good summary paper, looking at these metrics for multi-class problems: Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45, p. 427-437. (pdf) The abstract reads: This paper presents a systematic analysis of twenty four performance measures ...


28

The subset accuracy is indeed a harsh metric. To get a sense of how good or bad 0.29 is, some idea: look at how many labels you have an average for each sample look at the inter-annotator agreement, if available (if not, try yourself to see what subset accuracy the obtained when you are the classifier) think whether topic are well defined look at how many ...


23

To complement the other answers, here are some figures. One row = the expected output for one sample. Multiclass One column = one class (one-hot encoding) Multilabel One column = one class You see that: in the multilabel case, one sample might be assigned more than one class. in the multiclass case, there are more than 2 classes in total. As a side ...


22

For multi-label classification you have two ways to go First consider the following. $n$ is the number of examples. $Y_i$ is the ground truth label assignment of the $i^{th}$ example.. $x_i$ is the $i^{th}$ example. $h(x_i)$ is the predicted labels for the $i^{th}$ example. Example based The metrics are computed in a per datapoint manner. For each ...


18

Here is some discuss of coursera forum thread about confusion matrix and multi-class precision/recall measurement. The basic idea is to compute all precision and recall of all the classes, then average them to get a single real number measurement. Confusion matrix make it easy to compute precision and recall of a class. Below is some basic explain about ...


16

Here are some tips for enhancing the performance of VW models: Shuffle the data prior to training. Having a non-random ordering of your dataset can really mess VW up. You're already using multiple passes, which is good. Try also decaying the learning rate between passes, with --decay_learning_rate=.95. Play around with the learning rate. I've had cases ...


15

While there are some answers already on this forum I thought I'd give the explicit equations to make it more definite: Assuming you have a multi-class confusion matrix of the form, \begin{align} C=\text{Actual}\begin{matrix} & \text{Classifed} & \\ c_{11} & ... & c_{1n}\\ \vdots & \ddots & \\ c_{n1} & & c_{nn} \end{...


14

Using sklearn and numpy: from sklearn.metrics import confusion_matrix import numpy as np labels = ... predictions = ... cm = confusion_matrix(labels, predictions) recall = np.diag(cm) / np.sum(cm, axis = 1) precision = np.diag(cm) / np.sum(cm, axis = 0) To get overall measures of precision and recall, use then np.mean(recall) np.mean(precision)


13

Your Option 1 may not be the best way to go; if you want to have multiple binary classifiers try a strategy called One-vs-All. In One-vs-All you essentially have an expert binary classifier that is really good at recognizing one pattern from all the others, and the implementation strategy is typically cascaded. For example: if classifierNone says is ...


12

A likely cause is the fact you are not tuning your model. You need to find good values for $C$ and $\gamma$. In your case, the defaults turn out to be bad, which leads to trivial models that always yield a certain class. This is particularly common if one class has much more instances than the others. What is your class distribution? scikit-learn has ...


12

You can use a prior distribution over the classes. Let us assume that your model computes a vector of class probabilities $v$. You can define a vector of prior probabilities $\pi$ and then compute your class probabilities to be proportional to $v \circ \pi$, where $\circ$ denotes an element-wise product. So the probability that your observation belongs to ...


12

Definitions. In a classification task, your goal is to learn a mapping $h: X\rightarrow Y$ (with your favourite ML algorithm, e.g CNNs). We make two common distinctions: Binary vs multiclass: In binary classification, $\left|Y\right|=2$ (e.g, a positive category, and a negative category). In multiclass classifcation, $\left|Y\right|=k$ for some $k\in\...


11

Stratified sampling means that the class membership distribution is preserved in your KFold sampling. This doesn't make a lot of sense in the multilabel case where your target vector might have more than one label per observation. There are two possible interpretations of stratified in this sense. For $n$ labels where at least one of them is filled that ...


11

Based on the sentence you quoted, each item belongs to one class but can have several labels. Imagine you have animals like a fox, a chicken and a common European viper. A multi-class classification problem would be assigning them to a family: Fox Canidae Chicken Phasianidae Viper Viperidae In phylogeny, any species only has one ...


10

Unfortunately, because of the filter tree / elimination implementation in ECT, getting a measure of confidence is not straight-forward. If you can sacrifice some speed, using -oaa with logistic loss and the -r (--raw_predictions) option gives you raw scores that you can convert to a normalized measure of relative "confidence". Say you have a file like this ...


10

The problem does turn out to be parameter testing. I did not try when gamma is between 0.0 (which is 1/n_feature) and 1. On my data gamma should be turn to something around 1e-8


9

A multi-class problem has the assignment of instances to one of a finite, mutually-exclusive collection of classes. As in the example already given of crabs (from @Dikran): male-blue, female-blue, male-orange, female-orange. Each of these is exclusive of the others and taken together they are comprehensive. One form of a multi-label problem is to divide ...


8

Suppose we have data $(x_1, y_1), \dots, (x_k,y_k)$ where $x_i \in \mathbb{R}^n$ are input vectors and $y_i \in \{\text{red, blue, green} \}$ are the classifications. We know how to build a classifier for binary outcomes, so we do this three times: group the outcomes together, $\{\text{red, blue or green} \}$,$\{\text{blue, red or green} \}$ and $\{\text{...


8

Setup Recall that an SVM can be viewed as a weight vector $w$ and an intercept $b$, and that the output function for a test input $x$ is is $\langle w, x \rangle + b$. To get a binary prediction, we take $f(x) = \mathrm{sign}(\langle w, x \rangle + b)$. (I'm going to use some primal notations here, but use $\langle \cdot, \cdot \rangle$ to denote that ...


8

You cannot construct a ROC curve from the confusion matrix alone. A confusion matrix represents a single point in the ROC space, and you need all possible confusion matrices at all thresholds to build a full curve and compute the AUC. This holds true for multi-class ROC analysis.


8

I suposse that the Softmax function is applied when you request a probability prediction by calling the method mlp.predict_proba(X). To support my supposition I have developed this small experiment: from sklearn.neural_network import MLPClassifier from sklearn.datasets import load_iris import numpy as np X,Y = load_iris().data, load_iris().target mlp = ...


8

I think that the problem is not that you are using the classification methods poorly, but rather that this data has little predictive power for the regions. First of all, two classes have little data. table(y) East Asia & Pacific 35 Europe & Central Asia 56 Latin America & ...


8

The Gini impurity can definitely be used to quantify variance in a multi-class setting, not only in the binary case. Gini impurity is defined as $$ G(p) = \sum_{i=1}^{J}{p_i} \sum_{k \neq i}^{J}{p_k} = 1-\sum_{i=1}^{J}{(p_i)^{2}} $$ for the scenario with $J$ classes, each having probability $p_i...p_J$, where $|J|$ can be $>2$. More information can ...


7

Tensorflow has a loss function weighted_cross_entropy_with_logits, which can be used to give more weight to the 1's. So it should be applicable to a sparse multi-label classification setting like yours. From the documentation: This is like sigmoid_cross_entropy_with_logits() except that pos_weight, allows one to trade off recall and precision by up- or down-...


6

Some things you can try: Oversample your target classes. Insert duplicate records of your other three classes to augment your training dataset Undersample the negative responses. Instead of including all instances of other in your training data, only use a small portion. Bootstrap undersample the negative responses. This is probably your most robust option ...


6

Just to give a more clear understanding, I have explained each terminology with examples Multiclass classification/(One-Vs-One and One-Vs-ALL): Classification task with >2 classes! Assumption is that each sample is assigned to one and only one label E.g. MNIST E.g. a set of images of fruits which may be oranges, apples, or pears. Last ...


6

Yes, in general, you can. This approach you want to use is sometimes called "Micro-Averaging": first, sum all TNs, FPs, etc for each class, and then calculate the statistic of interest. Another way to combine the statistics for individual classes is to use so-called "Macro-Averaging": here you first calculate the statistics for individual classes (A vs not ...


Only top voted, non community-wiki answers of a minimum length are eligible