Questions tagged [log-loss]
The log-loss tag has no usage guidance.
59
questions
3
votes
1
answer
77
views
Selecting the model by bootstrapping: AIC vs. log-loss?
I'm building a predictive model with potentially multiple predictors. To that end, I try different, nested models, each with one more predictor than the previous one and compare their AICs. The AIC ...
- 7,969
1
vote
0
answers
16
views
Log base in Cross Entropy Loss [duplicate]
What is the base for the logarithm used in the cross entropy loss (while doing multiclass classification's backpropagation)? Is it e, 2, or 10?
- 111
2
votes
0
answers
45
views
Probabilistic interpretation for log-loss
Suppose I am modelling a binary response variable $Y \sim B(p)$ as a function of $p$ features $X_1, \dots, X_p$ by means of an equation of the form
$$
p(Y = 1 \, | \, X = x) = f(x,\theta),
$$
where $\...
17
votes
2
answers
2k
views
Why not use evaluation metrics as the loss function?
Most algorithms use their own loss function for optimization. But these loss functions are always different from metrics used for actual evaluation. For example, for building binary classification ...
- 887
3
votes
3
answers
173
views
Challenge an ICML Paper: For a given set of probability predictions and a log loss value, is the set of true labels giving such a loss unique?
Aggarwal's 2021 ICML paper "Label Inference Attacks from Log-loss Scores", seems to argue that the answer to the question in the title is "YES". The paper claims that, given ...
- 50.8k
1
vote
1
answer
951
views
Predicted Probability with XGBClassifier ranging only from 0.48 to 0.51 for either class
Why does my XGBClassifier predicts probability only from 0.48 to 0.51 for either class?
I'm very new to XGBoost, so any suggestions are greatly appreciated! Here's ...
- 113
5
votes
2
answers
902
views
"Dumb" log-loss for a binary classifier
I am trying to understand how I can best compare a classifier that I have trained and tuned against a "dumb" classifier, particularly in the context of binary classification with imbalanced ...
- 53
1
vote
1
answer
209
views
Mean logarithmic square error
I have a fallowing problems: I'm training a neural network against some set of output values (regression problem). Those values are between -inf to inf and I can't normalize them, because they come ...
- 557
3
votes
1
answer
177
views
exp(log_softmax) vs softmax as neural network activation
I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So ...
1
vote
0
answers
34
views
Is half the log loss, twice as good?
Lets say I have two different models based on the same dataset.
Model A has a log loss of 0.30 on this dataset.
Model B has a log loss of 0.60 on this dataset.
If our scoring metric is log loss, is ...
- 223
1
vote
1
answer
726
views
Understanding multiclass log-loss
I'm trying to understand the multiclass log-loss as described in sk-learns documentation.
The wording ' Let the true labels (Y) for a set of samples be encoded as a 1-of-K binary indicator matrix...', ...
- 569
0
votes
0
answers
121
views
If you're trying to match a vector $p$ to $x$, why doesn't a divisive loss function $\frac{p}{x} + \frac{x}{p}$ work better than negative log loss? [duplicate]
Suppose you had a classification problem where you are trying to predict a class label (e.g., $[0 \: 1 \: 0]^T$) with a model. One way to do this is to use log loss:
$\Large \ell_{\log} = -\sum_i[y_i\...
- 195
1
vote
1
answer
730
views
F1 weighted vs. Log loss in SciKit learn RandomSearchCV
I am sorry to ask another question regarding this topic but I am still puzzled about the following:
When I use 'F1_weighted' as my scoring argument in a ...
- 171
0
votes
0
answers
9
views
testing the accuracy of a probability logistic regression [duplicate]
I had a dataset with binary data. I created a logistic regression model with continuous data on the x-axis and binary data that has values 0 and 1 on the y-axis. Then I plotted my model, a line that ...
- 35
1
vote
1
answer
356
views
Proof that log-odds minimize binomial deviance
How do you prove that minimizing the binomial deviance estimates the log-odds? i.e:
$$
\ln{\left ( \frac{p(x_i)}{1-p(x_i)} \right )} = \underset{f(x_i)}{argmin} \ \mathbb{E} \left [y_i \ln \left ( \...
- 1,021
12
votes
2
answers
2k
views
Reference for log-loss (cross-entropy)?
I'm trying to track down the original reference for the logarithmic loss (logarithmic scoring rule, cross-entropy), usually defined as:
$$L_{log}=y_{true} \log(p) + (1-y_{true}) \log(1-p)$$
For the ...
- 3,420
1
vote
1
answer
308
views
Are maximizing the log probability and assigning the ground-truth token the highest rank the same?
I have been reading this paper titled Neural Text Generation with Unlikelihood Training. It is about the maximum likelihood function used to train generative models. Anyway, it says that a major flaw ...
- 113
3
votes
2
answers
5k
views
Understanding cross entropy loss [closed]
The formula for cross entropy loss is this:
$$-\sum_iy_i \ln\left(\hat{y}_i\right).$$
My question is, what is the minimum and maximum value for cross entropy loss, given that there is a negative sign ...
- 33
2
votes
2
answers
512
views
Classification Models giving probabilities at extreme end only
I am building a binary classification model with proportion of 1 is at only 3% and total 70000 data points.I have 5 variables out of which 3 are coming out to be important.
I have built model using ...
- 113
8
votes
2
answers
2k
views
Logarithmic loss vs Brier score vs AUC score
I have a dataset with two classes of elements. I also have two methods which assign (complementary) probabilities to each element in the dataset of belonging to either class.
Given that I work with ...
- 3,420
0
votes
1
answer
273
views
Negative Log-Likelihood Loss with Gibbs distribution for beta approaching infinity
This might be more of a math question, but since it concerns ML, I'll ask it here.
In "A tutorial on energy-based learning" (LeCun et al., 2006), on page 15, section 2.2.4 about the Negative Log-...
- 11
1
vote
2
answers
715
views
Use of sigmoidal function in output layer to predict continuous values and correct use of the loss function
Lets supose that a neural network is utilized to map a set of training data to a continuous interval between 0 and 1 utilizing a sigmoidal function on its output layer. Is it correct to optimize the ...
- 71
1
vote
0
answers
175
views
binary cross entropy vs multi cross entropy
i am new to neural networks
I know that multi class entropy is same as binary class entropy when
the categories are only (0,1), but can some one explain it
mathematically with an example that ...
- 111
3
votes
3
answers
452
views
How to compare log-loss across similar classification models with different baseline probabilities?
Suppose I have two datasets, A and B, which share a feature vector $X$ but have different units of analysis (e.g. people from two different countries). I have trained classifiers with the same model ...
- 217
3
votes
1
answer
1k
views
Why Log loss, AUC and precision & recall change differently when class imbalance problem is tackled?
I have a dataset and I'm working on a binary classification task with it. It has a class imbalance problem where False class versus ...
- 2,313
1
vote
1
answer
2k
views
How does Cross-Entropy (log loss) work with backpropagation?
I am having some trouble understanding how Cross Entropy would work with backpropagation.
For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of ...
- 11
1
vote
0
answers
669
views
Does probability calibration always enhance the logloss?
I'm trying to calibrate some probabilities returned by different classifiers. I have plotted the auc and logloss of each one and ...
- 882
3
votes
0
answers
297
views
derivative of cross entropy yields log-odds, does that make sense?
I am looking for a proof how to derive the logistic regression from cross-entropy loss, i.e. derive the form of a sigmoid from cross entropy. my thoughts are these:
$\ell = y_i \ln{p_i} + (1-y_i)\ln{...
- 743
0
votes
0
answers
22
views
Is this expression for Loss valid?
The negative likelihood loss over training set $t$ where a training instance is given by $x^{(t)}$, taget by $y^{(t)}$, a specific masking (dropout) by $m$ and weights by $w$ as:
$$L(w|m) = -\sum_t ...
- 215
1
vote
0
answers
27
views
How does one go about evaluating a new loss function? [closed]
Suppose a new loss function similar to log loss is proposed. Now we have to check whether it'll work in practical scenarios in general and tractable. What experiments one should run get to some ...
- 11
5
votes
1
answer
2k
views
Pytorch Cross Entropy Loss implementation counterintuitive
there is something I don't understand in the PyTorch implementation of Cross Entropy Loss.
As far as I understand, theoretical Cross Entropy Loss is taking log-softmax probabilities and output a real ...
14
votes
4
answers
5k
views
Why is binary cross entropy (or log loss) used in autoencoders for non-binary data
I am working on an autoencoder for non-binary data ranging in [0,1] and while I was exploring existing solutions I noticed that many people (e.g., the keras ...
- 193
1
vote
0
answers
156
views
How is the minimum logarithmic loss calculated when initializing the XGBoost algorithm?
Suppose there are $5$ sample units, $2$ of which carry the feature $y=1$ to be predicted and three of which carry the feature $y=0$. So, $2$ are positive.
The XGBoost algorithm initializes with
$\...
- 11
1
vote
1
answer
2k
views
How to achieve variational autoencoder (VAE) with unrestricted input?
For a normal VAE an input and a reconstruction with values in the range of $[0, 1]$ are expected.
This is necessary since the log loss only makes sense for this range.
If the input is not within $[0, ...
- 141
1
vote
0
answers
268
views
Binary cross etropy loss with non binary ground truth data [closed]
Is it possible to use binary cross etropy loss with non binary ground truth data, i.e. not [0,1] values, but [0,0.1,0.5,1.0] ...
- 2,107
2
votes
1
answer
887
views
Intuition behind logloss function
I have a difficulty understanding the intuition behind the logloss function since it seems to totally ignore negative examples where y = 0.
The images below visualize my question to some extend:
...
- 245
6
votes
2
answers
4k
views
Calculate binomial deviance (binomial log-likelihood) in the test dataset
I'm predicting probabilities $\mathbb{P}(Y=1)$ using a probability forest (ranger in R). I want to evaluate my predictions $\hat ...
- 81
1
vote
0
answers
173
views
Describing relative performance of models on the basis on log-loss / cross-entropy
Suppose I train classification algorithms on a given dataset with three model specifications: A, B, and C. On validation data, model A has an average log-loss of 1.0, model B has a log-loss of 0.75, ...
- 217
0
votes
0
answers
1k
views
How to minimize this loss function?
I have a loss function
$L = \sum_i I(\text{log_loss}_i > 0.6)$, where $I$ is the indicator function (equals 0 when argument is false, 1 otherwise) and log_loss$_i$ is the log loss of the $i^{th}$ ...
- 193
1
vote
0
answers
315
views
Log loss when classes are -1 and +1 [duplicate]
We can calculate the log loss for a classification problem with two classes as follows:
where y is the label of the actual class and ...
- 537
0
votes
0
answers
199
views
Difference between using thresholds to classify and using a custom objective function
Suppose we are faced with a binary classification problem. The standard approach seems to be to fit a probability estimator using a loss function like log loss and then afterwards determine the ...
- 43
4
votes
1
answer
2k
views
Is there a cross-entropy-like loss function for multiple classes where misclassification costs are not identical?
For this conversation I'll use the below definition of cross-entropy where there are N samples, M different classes, $ y_{ij} $ is 1 if sample i is of class j and 0 otherwise and $p_{ij}$ is the ...
- 43
1
vote
0
answers
887
views
Gradient boosting for binary outcome - terminal nodes estimate (using R gbm)
I have been searching for the answer for the below query quite a long time and found a few answers (see:
interpretation of gbm single tree prediction in pretty.gbm.tree
or R Package GBM - Bernoulli ...
- 21
1
vote
1
answer
577
views
Interpreting Inf (Infinity) as answer to logloss function?
so I am using this logloss function
logLoss = function(pred, actual){
-1*mean(log(pred[model.matrix(~ actual + 0) - pred > 0]))
}
sometimes it is correctly ...
19
votes
1
answer
18k
views
logloss vs gini/auc
I've trained two models (binary classifiers using h2o AutoML) and I want to select one to use. I have the following results:
...
- 1,408
5
votes
1
answer
12k
views
Log Loss function in scikit-learn returns different values
I have been trying to wrap my head around the log loss function for model evaluation. I understand how the value is calculated after doing the math by hand.
In the python module ...
- 63
5
votes
1
answer
686
views
Smoothing/shrinking the predicted probability of a classifier to reduce live logloss
Let us assume we work on a 2 -class classification problem. In my setting the sample is balanced. To be precise it is a financial markets setting where up and down have approximately 50:50 chance.
The ...
- 3,326
2
votes
1
answer
1k
views
Probability Calibration messes Reliability
I have about 1000 samples with 20 features and I'm using Random Forest to predict a binary class.
I'm trying to apply the probability calibration process as described on scikit using ...
- 191
1
vote
1
answer
483
views
Sudden increase in log-loss on training set with H2O GBM
I am performing a hyperparameter grid search for a GBM classifier in H2O, running version 3.10.4.8 (on top of python 3.5.3). This is a multiclass problem (~40 classes).
As a first test, I tried a ...
2
votes
2
answers
4k
views
Performance Metrics for Imbalanced Classification
I'm trying to fit multiple Stochastic Gradient Descent models to a dataset where the target (binary target, 0 or 1) is very imbalanced, i.e the success rate is about 0.0001.
Out of all the models I'...
- 997