Questions tagged [confusion-matrix]

A confusion matrix is a contingency table used to evaluate the predictive accuracy of a classifier. Confusion matrix is the 2x2 frequency table with counts "True positive", "True negative", "False positive", "False negative", relating classifying to a class of interest vs. else class. But in a broader sense, any frequency kxk crosstabulation "Predicted" x "Actual" classes can be called a confusion matrix, in the context of evaluation of a classifier.

Filter by
Sorted by
Tagged with
0 votes
0 answers
27 views

When are we satisfied with the performance of a classifier?

I am training a multi-head classifier (based on Resnet18) for a multilabel classification task. The dataset I am using to train the classifier is noisy and to improve the accuracy for the worse-...
user avatar
  • 137
0 votes
0 answers
8 views

Calculating entropy-based indices based on a frequency cross-tabulation of objects in Python

I was wondering whether there exist a Python solution for calculating Mutual Information, Completeness, V-Measure, etc., based on a pair-confusion matrix. MI defined as H(C_0) + H(C_1) - H(C_O&C_1)...
user avatar
  • 61
0 votes
0 answers
37 views

Are Type I and Type II errors in multiclass problem appear to be the same?

As far as I know from binary classification FP error is a type 1 error FN error is a type 2 error I have this confusion matrix generated: And here I found how to read this confusion matrix: As you ...
user avatar
  • 101
1 vote
0 answers
23 views

How to assign costs to the confusion matrix

I am trying to assign costs to the confusion matrix. That is, in my problem, a false positive (FP) does not have the same cost as a false negative (FN), so I want to assign to these cases a cost "...
user avatar
2 votes
1 answer
62 views

Two confusion matrices

In a machine learning context, I am working on a binary classification problem. There is a source of truth $T$ for labels, and a labeling process $A$ which is not perfect and makes errors compared to $...
user avatar
  • 1,387
0 votes
0 answers
27 views

Confusion matrix vs ROC-AUC curve

I'm using the lares packages in combination with H2O and with ...
user avatar
  • 2,207
0 votes
0 answers
15 views

Understand AUCs of different models

I'm testing two models against each other: provides an AUC of 93,94 % with TP = 99,9 %, TN = 0 %, FP = 100 % and FN = 0.1 %. provides an AUC of 92,78 % with TP = 98,8 %, a TN = 30,6 %, FP = 69,4 % ...
user avatar
  • 2,207
0 votes
0 answers
23 views

Make ROC graph with my problematic [duplicate]

For a personal project, i'm trying to figure out how to trace some ROC/AUC graph with my current problematic. I have a list of thunder flashes, and i'm trying to find if a meteorological variable (...
user avatar
0 votes
1 answer
21 views

How to interpret this confusion matrix and roc curve?

I got these two images for evaluating a RF: I wonder why the ROC curve seems to be so good while the confusion matrix shows that the True Positive isn't so good with only ~16 %? By looking only at ...
user avatar
  • 2,207
0 votes
0 answers
27 views

Interpreting puzzling result from a confusion matrix from two different algorithms

I use a data set from Kaggle on Home Credit Default Risk to learn more about classification algorithms. If I run two very basic algorithms I get puzzling result from the confusion matrix. The logistic ...
user avatar
  • 101
1 vote
1 answer
71 views

Logistic Regression: What is the value for precision when recall (true positive rate) is 0?

A quick overview of definitions before I get into the question: True Positive (TP): An actual positive that the model classified as positive False Positive (FP): An actual negative that the model ...
user avatar
  • 211
0 votes
0 answers
28 views

Does a single F-score make sense for multiclass classification?

I'm newish to ML and trying to compute some performance metrics for a K-means classifier. I've constructed the following confusion matrix, and computed the per-class precision, recall, and F-scores (...
user avatar
1 vote
0 answers
38 views

Confusion matrix error [closed]

I have been trying to do support vector machine classification in R and confusion matrix has been showing the error even after changing into factors confusionMatrix(as.factor(yTestPred), as.factor(...
user avatar
  • 11
3 votes
1 answer
154 views

why I am getting TP and FP in confusion matrix is 0, how to get it right?

why I am getting TP and FP in confusion matrix is 0, how to get it right? tp = 0,fp = 0,tn = 9847,fn = 18
user avatar
1 vote
1 answer
23 views

Estimate sample size from a variable population

Context: I want to measure accuracy, precision, and recall for individual raters. Each rater completes a variable amount of labels, for ex. rater A may complete 500 in a given time period while rater ...
user avatar
1 vote
1 answer
193 views

Creating confusion matrix and calculating TP, TN,FP, FN

I want to generate a confusion matrix for class 2. There is a total of four input classes namely 0,1,2,3. In particular test data batch, there is no input ...
user avatar
0 votes
2 answers
307 views

My training accuracy is 1.0 and my test accuracy is 0.994. Am I overfitting for a multiclass classification?

This is a multiclass classification for an imbalanced dataset. I set the class_weight for this model to "balanced". I have a perfect training accuracy (1.0) and a nearly perfect testing ...
user avatar
0 votes
0 answers
176 views

understanding confusionMatrix output in R

My PI wants me to run a confusion matrix on two variables that are 0/1 and should (but probably aren't) reasonably close. I have never run a confusion matrix before and have only a limited idea of ...
user avatar
  • 329
1 vote
1 answer
67 views

How to compare ctree and cforest approaches on the classification task?

I doing some numerical experiments with ctree() and cforest() functions from the partkid package. I am using the Wine Quality Data Set. ...
user avatar
  • 840
0 votes
1 answer
19 views

In confusion matrix prediction, has the training data used for prediction been seen by the model

I understand that Confusion Matrix evaluates prediction results with metrics such as precision, recall, accuracy and F1-Score. My question is regarding the data used for prediction. Is it okay if this ...
user avatar
2 votes
1 answer
35 views

When comparing classifiers on different datasets with different prevalences, is it valid to calculate prevalence-adjusted PPV?

Scenario: comparison of 2 different binary classifiers Both classifiers report sensitivity and specificity and number actually positive (P), but classifier 1 is tested on a dataset with prevalence 20%,...
user avatar
0 votes
1 answer
99 views

Why does confusion matrix shows inverse output?

I am working on a logistic regression binary classification with 1000 rows and 28 columns While I have split my dataset into train and ...
user avatar
  • 1,872
0 votes
0 answers
13 views

Likelihood of True Positive only from TPR and FPR

When preparing for an interview, I got the following question (on this medium link), Given a medical test produce a 1% False Positive rate (FPR), and the population True Positive Rate (TPR) is 5%, ...
user avatar
  • 103
0 votes
0 answers
140 views

How to interpret a confusion matrix in a multi-class prediction task?

I am struggling to interpret a confusion matrix in a multi-class prediction task. I have the data in three classes: 'A', 'B', 'C', and I used SVC to do the classification. In order to see the ...
user avatar
0 votes
0 answers
16 views

Validation and test metric with k-fold CV

I am following this procedure from sklearn page: I got a dataset with 42000 samples, and i have used train_test_split to split ...
user avatar
  • 101
0 votes
0 answers
96 views

Can you calculate recall/precision/f1 score for "continuous variables"

My goal is to calculate some performance metrics (precision, recall, f1,accuracy) on a binary variable (yes or no) but have dates associated with each data point. Attaching dates to this binary ...
user avatar
  • 1
1 vote
0 answers
40 views

Calculating overall Precision and Recall, one versus all

I've got several confusion matrices, all of binary classification (negative, positive). I would like to get general scores of all the matrices combine. Problem is, that the data is not balanced at all....
user avatar
  • 11
1 vote
1 answer
37 views

How to fully evaluate a multiclass classification problem?

When you have a multiclass classification problem, what is the right way to evaluate it's performance? What I usually do is to display the confusion matrix and the ...
user avatar
0 votes
1 answer
41 views

How to practically calculate the accuracy of each class in muliclass classification problem?

I have the following confusion matrix: ...
user avatar
0 votes
0 answers
23 views

Difference in model result when scoring data

I am trying to predict/score on an out-of-time data applying the same model I had used to train and predict on test data. In the previous model, the recall and precision for correctly predicting 1s ...
user avatar
2 votes
2 answers
484 views

How can accuracy be greater than my precision, recall and F-Score metrics?

I have trained two models to detect gestures using ambient light and solar panels. I am now testing the two models in different light scenarios. I have a Convolutional Neural Network model that ...
user avatar
  • 21
0 votes
0 answers
44 views

In multi-class, is the average accuracy of each class in confusion metrics equal to the accuracy calculated from cross validation?

When I calculate accuracy through cross validation, it gives me a different accuracy than when I calculate through confusion metrics. Why does it give different accuracy? Is overall calculated ...
user avatar
1 vote
1 answer
175 views

Interpreting NaN values for precision in Confusion Matrix

Please refer to the confusion matrix here: https://imgur.com/a/Iq1epre Would I get precision values of NaN because of 0/0 in the right most columns? Is that even possible? How should I interpret this? ...
user avatar
  • 47
1 vote
1 answer
72 views

How to obtain conditional use accuracy equality with communities with different real positive rates?

The short version is that I would like to know what the confusion matrices (numbers of true positives, false positives, true negatives, and false negatives) should be to achieve conditional use ...
user avatar
0 votes
0 answers
16 views

Testing a churn model trained on fix period data

A Churn model that needs to predict the customers more likely to churn after 9 months (from the executon time),the model was trained on customer's (retail bank) transactions/ loans 3 years data ...
user avatar
0 votes
0 answers
28 views

Is there a F1-score for the negative class?

I am analysing some data and calculated the $F1 = \frac{2TP}{2TP + FP + FN}$ I'd like to know if there is a name to a negative-class based F-score, something like $F0 = \frac{2TN}{2TN + FP + FN} = 2 \...
user avatar
0 votes
0 answers
40 views

A question about precision

We have 2 Classification models (Random forest on balanced Data Set), the first one classify a Bank's client as a Churner (closing his account) or an active client, the second one classify a Bank's ...
user avatar
2 votes
2 answers
330 views

How to compute a 'pair confusion matrix'?

I don't really understand how the pair confusion matrix (used for example in comparing of clusterings) is calculated... ...
user avatar
  • 23
0 votes
1 answer
83 views

Should train and test dataset label distribution match reality?

Assume I’m building a binary classification model to detect a disease that according to research is present in 3% of the population. I’m given a balanced dataset to train the model where half of the ...
user avatar
  • 275
0 votes
1 answer
58 views

Interpretation of number of observations in confusion matrix after k-fold cross validation

I completed a 5-fold cross validation and produced a confusion matrix that (I believe) summarizes the results of the 5-fold cv; however, I don't understand why there are only 8 observations in the ...
user avatar
  • 19
1 vote
1 answer
734 views

Creating a confusion matrix for every fold from k-fold cross validation in R

I am attempting to run a 5-fold cross validation in R for a logistic regression model followed up by confusion matrices for each "fold." However, my code is only producing one confusion ...
user avatar
  • 19
0 votes
0 answers
14 views

How should I calculate recall and precision on unbalanced data?

I have a model that classifies people into three segments (X, Y, Z). The model says that there are 150 of X, 250 of Y, and 5000 of Z. However, I don't know for certain that is the overall prevalence ...
user avatar
  • 156
0 votes
1 answer
140 views

Why is it called Sensitivity/Recall and Specificity?

Where do the terms: Sensitivity, Recall and Specificity come from historically? I've been looking for an answer for quite some time but to no avail. I understand the formulae and what they mean but I ...
user avatar
  • 139
0 votes
0 answers
15 views

How do I analyse my confusion matrix?

How do I analyze my confusion matrix outcome in terms of FP/TP/FN/TN, precision, etc? ...
user avatar
  • 11
0 votes
0 answers
11 views

Training and Testing accuracy High, but actual testing low

I have approximately 377 samples of an audio event A and the exact amount of another event B. I am using a 10 fold cross validation method to classify the binary event. My Training accuracy comes in ...
user avatar
  • 125
0 votes
0 answers
214 views

What is no information rate in confusionMatrix in r? [duplicate]

I wrote a logistic regression and used confusionMatrix function of caret library to check the accuracy of this model. Here is my code. ...
user avatar
2 votes
0 answers
139 views

Multi-class optimal threshold selection

What are the methods typically used for optimal threshold selection of multiclass classification problem ? I implemented the detector/classifier and generated the confusion matrix, but would like to ...
user avatar
0 votes
0 answers
31 views

Why on a Confusion Matrix are Independent and Dependent Axis Transposed?

Normally, a graph's X axis contains the independent values and the Y axis contains the dependent values. However, on a Confusion Matrix, the X is dependent (predicted) and Y is independent (true). Why ...
user avatar
  • 103
7 votes
2 answers
325 views

Is there something like a confusion matrix for a probabilistic score rather than categories?

Imagine we have pictures of three animals: dogs, cats, and horses. We train our image classifier and get a confusion matrix, noticing that the model tends to predict that dogs are horses. But then we ...
user avatar
  • 32.4k
0 votes
0 answers
40 views

Confusion matrix test-retest reliability (McNemar or paired t-test)

I would like to ensure test retest reliability between test runs. Each run produces a 2x2 confusion matrix reporting the number of true positives, false positives, false negatives, and true negatives. ...
user avatar

1
2 3 4 5 6