Linked Questions
100 questions linked to/from Proper scoring rule when there is a decision to make (e.g. spam vs ham email)
1
vote
1
answer
492
views
How would you find a p threshold for a binary classification prediction? [duplicate]
Lets say that there's a binary classification problem where $X$ ∈ $R_p$ and $Y ∈ \{0,1\} $ and $Pr(Y = 1 | X = x) = p$ for $p$ in $[0,1]$. There is a loss function $L_{falseneg} > 0$ for false ...
- 11
204
votes
10
answers
105k
views
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...
Tim♦
- 117k
75
votes
4
answers
51k
views
Reduce Classification Probability Threshold
I have a question regarding classification in general. Let $f$ be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if $P(c|D) > 0.5$, we will ...
- 1,259
24
votes
1
answer
3k
views
What is the importance of probabilistic machine learning?
I am attending a course on "Introduction to Machine Learning" where a large portion of this course to my surprise has a probabilistic approach to machine learning (ML), like a probabilistic ...
- 421
14
votes
3
answers
2k
views
How would I bias my binary classifier to prefer false positive errors over false negatives?
I've put together a binary classifier using Keras' Sequential model. Of its errors, it predicts with false negatives more frequently than false positives.
This tool is for medical application, where I'...
- 143
30
votes
2
answers
5k
views
Choosing among proper scoring rules
Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences ...
- 5,478
17
votes
3
answers
2k
views
Brier Score and extreme class imbalance
Since I've heard about proper scoring rules for binary classification like the Brier score or Log Loss, I am more and more convinced that they are drastically underrepresented in practice in favor of ...
- 173
8
votes
2
answers
8k
views
Best way to reduce false positive of binary classification to exactly 0?
I'm working on a task that even a 0.00001 fp rate is not acceptable, because detecting something as a positive when its not will have very bad consequences in this task, so it needs to be exactly 0 ...
- 225
16
votes
2
answers
897
views
Why not use evaluation metrics as the loss function?
Most algorithms use their own loss function for optimization. But these loss functions are always different from metrics used for actual evaluation. For example, for building binary classification ...
- 835
5
votes
3
answers
580
views
Interpretation of coefficients in a poorly performing GLM
Suppose that I have trained a logistic regression model on a certain dataset, and I wish to interpret the coefficients of this model.
Does it make any difference on the validity of the interpretation ...
- 61
14
votes
1
answer
1k
views
Why is LogLoss preferred over other proper scoring rules?
It seems anytime people care about estimating probabilities accurately they choose LogLoss as the evaluation metric. But there are many other evaluation metrics which will prefer accurate estimation ...
- 320
13
votes
3
answers
435
views
How do you know that your classifier is suffering from class imbalance?
In cases where there is a substantial difference in relative class frequencies, it could be that the density of the minority class is never higher than the density of the majority class anywhere in ...
- 49.4k
4
votes
2
answers
331
views
Prediction using a logistic regression model
Given a logistic regression model:
$y \in \{0, 1\}$
$ P(y=1|x;\theta) = h_{\theta}(x) = \frac{1}{1+\exp(-\theta^T x)}$
And given the value $\theta^*$ which maximises the conditional likelihood $P(y|X; ...
- 151
0
votes
2
answers
2k
views
What is the loss function for regression using artificial neural network?
For classification, one of the most common loss functions for artificial neural networks (ANN) is cross-entropy. What about in ANN for regression?
and why is cross-entropy hardly even discussed in ANN'...
- 3,139
5
votes
2
answers
313
views
Which class to define as positive in unbalanced classification
In a classification task, why do we usually choose the minority class as "positive" case for response variable? For example, if there are 1000, 9000 for class A and B respectively, we ...
- 387