0
$\begingroup$

I am trying to understand precision and recall for multiclass classifications. For the concept of True Positive or True Negative, the model is classifying This vs. That and not necessarily Yes vs. No if that makes sense. However, in my custom cross-validation all of the predictions should be the same class if the model was 100% accurate (i.e. protein-synthesis). I may be over thinking this but I'm trying to figure out which average parameter to use and whether or not I should use the label argument.

My classes are largely imbalanced:

cell-wall               232
dna-synthesis           192
protein-synthesis       114
rna-polymerase           76
fatty-acid-synthesis     36
cell-membrane            33

Which makes me think that I should be using average='micro' based on the documentation: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html

average: [None, ‘binary’ (default), ‘micro’, ‘macro’, ‘samples’, ‘weighted’] This parameter is required for multiclass/multilabel targets. If None, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data:

'binary': Only report results for the class specified by pos_label. This is applicable only if targets (y_{true,pred}) are binary.

'micro': Calculate metrics globally by counting the total true positives, false negatives and false positives.

'macro': Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

'weighted': Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.

'samples': Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).


My questions:

  1. When I set average='micro' for precision_score it turns out to be the same as my accuracy_score and it becomes 1.0 when I set label=['protein-synthesis']. I don't understand why the value would be 1.0. I understand that it is only considering predictions of that label, but wouldn't this always be 1.0? Is this value of 1.0 meaningful or would be more productive to just set average='micro' and get the same value as my accuracy_score`?**

  2. When I set average='macro' for both precision_score and recall_score the value seems to be the average of 0 and the output when average='macro' and label='protein-synthesis'. Why would anyone want this? Is this for situations when I would expect to get predictions that are more than one class (for example, if my model was not as accurate)?**

  3. What type of situation would accuracy be different than precision or recall in this scenario?


import numpy as np
from sklearn.metrics import accuracy_score, precision_score, precision_score

# Prediction
y_pred = np.array(['dna-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis'], dtype=object)
y_true = np.array(['protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis', 'protein-synthesis'], dtype=object)
classes = ['cell-membrane', 'cell-wall', 'dna-synthesis', 'fatty-acid-synthesis', 'protein-synthesis', 'rna-polymerase']

print("Accuracy", accuracy_score(y_true=y_true, y_pred=y_pred), sep="\t", end=2*"\n")
# Accuracy: 0.9666666666666667

for average in ["micro", "macro"]:
    print("Precision", f"average={average}", precision_score(y_true, y_pred, average=average), sep="\t")
    print("Precision", f"average={average}|label=[{y_true[0]}]", precision_score(y_true, y_pred, average=average, labels=[y_true[0]]), sep="\t")
print()
# Precision average=micro   0.9666666666666667
# Precision average=micro|label=[protein-synthesis] 1.0
# Precision average=macro   0.5
# Precision average=macro|label=[protein-synthesis] 1.0

for average in [ "micro", "macro"]:
    print("Recall", f"average={average}", recall_score(y_true, y_pred, average=average), sep="\t")
    print("Recall", f"average={average}|label=[{y_true[0]}]", recall_score(y_true, y_pred, average=average, labels=[y_true[0]]), sep="\t")
# Recall    average=micro   0.9666666666666667
# Recall    average=micro|label=[protein-synthesis] 0.9666666666666667
# Recall    average=macro   0.48333333333333334
# Recall    average=macro|label=[protein-synthesis] 0.9666666666666667
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.