A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

learn more… | top users | synonyms

1
vote
1answer
28 views

derivative of loss function

If we have a paralyzed loss function of the form of: \begin{align} L(\beta)& =\frac{1}{2}(y-X\beta)^T(y-X\beta)+ \lambda \beta^T f(\beta) \end{align} where $X_{n\times m}$ and $\beta_{m \times ...
2
votes
1answer
53 views

Scikit Binomial Deviance Loss Function

This is scikit GradientBoosting's binomial deviance loss function, ...
0
votes
0answers
28 views

What is the activation function, label and loss function for Hierachical Softmax [migrated]

Several papers([1],[2], [3]) suggest the use of Hierachical Softmax instead of softmax for classification where the number of classes is large (eg many thousand). I haven't been able to get clear in ...
1
vote
3answers
164 views

Best loss function for very sparse real-valued data

Suppose the target output of my data prediction model is an $M\times N$ matrix where $95\%$ of the values are $0.0$ and the other values are anywhere between $0.0$ and $1.0$, what would be a good loss ...
0
votes
0answers
28 views

Gradient for hinge loss multiclass

I am little confused when trying to find the gradient for the multiclass hinge loss: $l(y) = \max( 0, 1 + \underset{r \neq y_i}{ \text{max} \ } W_r \cdot x_i + W_{y_i} \cdot x_i)$ Where $W^{k \times ...
0
votes
0answers
53 views

Loss and dropout in deep learning

I have a CNN with 3 convolutional layers, 1 max-pooling layer and 2 fully-connected layers before applying softmax classification. The CNN is trained with Adagrad and I achieve a quite good ...
0
votes
0answers
20 views

Logistic loss approximation

In many implementations of logistic loss (example sklearn) I see the following code(adapted from sklearn), where p is the prediction and y the true value: ...
1
vote
1answer
46 views

Gradient of loss function for (non)-linear prediction functions

$ \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\xv}{\mathbf{x}} \newcommand{\loss}{L(\wv;\xv, y)} $ I'm trying to clear up the calculation of the gradient of a loss function, ...
0
votes
0answers
27 views

Meaning of the prior and loss parameters in rpart in R

Could someone please explain to me what specifying priors and/or loss parameters in R's rpart actually do? I found R's documentation completely unhelpful. For example, let's suppose I have a highly ...
1
vote
0answers
34 views

hinge loss vs logistic loss advantages and disadvantages/limitations

Hinge loss can be defined using $\text{max}(0, 1-y_i\mathbf{w}^T\mathbf{x}_i)$ and the log loss can be defined as $\text{log}(1 + \exp(-y_i\mathbf{w}^T\mathbf{x}_i))$ I have the following questions: ...
0
votes
0answers
27 views

maximum mis-classification loss of hinge loss

In the plots and in some lecture notes, I read that hinge loss is bounded between (0,2). But I can not understand that. By definition, hinge loss is (standard one) ...
4
votes
1answer
64 views

showing that $\bar{X}$ is inadmissible by comparing with $\max(\bar{X},2)$ under squared error loss function

suppose $X_1,X_2,\ldots,X_n$ be a random sample of $N(\theta,1), \theta>2$. how can I show $\bar{X}$ is inadmissible estimator Compared to $\max(\bar{X},2)$ under Squared error loss function
2
votes
0answers
31 views

Does maximum likelihood minimize a kind of generalized “0-1 loss”?

A very good point was raised here about how the optimal betting strategy under 0-1 loss was to bet on the mode, while under MSE loss the optimal strategy was to bet on the mean. Maximum likelihood ...
0
votes
3answers
52 views

Equating the two equations of Ridge Regression

I'm studying Ridge Regression now and I'm having a bit of trouble understanding how to relate the two equations that pop up when I read about it. There is the coefficient estimate: $$\hat{\beta} = ...
7
votes
1answer
108 views

What would be an example of when L2 is a good loss function for computing a posterior loss?

L2 loss, together with L0 and L1 loss, are three a very common "default" loss functions used when summarising a posterior by the minimum posterior expected loss. One reason for this is perhaps that ...
0
votes
0answers
72 views

Implementing WARP Loss (Gradient Computation)

I am trying to implement the WARP Loss in Torch, as defined in the WSABIE paper: http://www.thespermwhale.com/jaseweston/papers/wsabie-ijcai.pdf The Algorithm is as follows: The Algorithm ...
0
votes
0answers
59 views

Value of the loss function and Classification Errors in gbm package (R)

I have a simple problem of classification (0s and 1s) using adaboost loss function. When I check the components of a boosted model using the gbm package I see: ...
1
vote
0answers
34 views

Loss function from Posterior log likelihood

I am trying to implement Bayesian Logistic Regression and was looking at this paper http://www.stat.columbia.edu/~madigan/PAPERS/techno.pdf. The authors in this paper had assumed the $\beta$s to have ...
1
vote
0answers
24 views

Loss function for rank deficient covariance matrices?

I'm trying to compare the efficiency of different estimators of the covariance matrix of a particular type of multivariate normally distributed data. This comparison, as well as the estimation process ...
2
votes
1answer
90 views

Get distribution for aggregate loss using Monte Carlo

I am given two data sets containing dates and losses (in some currency). Given a distribution for the amount of losses and an (a,b,0) distribution for frequency of losses, how can I use Monte Carlo ...
3
votes
1answer
74 views

Designing Asymmetric regression (assymettric loss for regression)

I have a hybrid classification/regression problem.The predicted value can be assumed to be centred around 0. I want to penalize the predictor more, if the predicted value and actual value have ...
2
votes
1answer
109 views

minimax property of sample mean

Suppose $X_1,X_2,\ldots,X_n$ are iid $\mathcal{N}(\mu,\sigma^2)$, where $\sigma$ is known, but $\mu$ is not. We wish to construct a confidence interval of length $L$ (given) for $\mu$. Is it true that ...
0
votes
1answer
36 views

Estimating conditional variance y|x

I am building a predictor for $y = f(x)$ using training samples ${(x_i, y_i)}$ (assume) drawn i.i.d from some distribution $p(x,y)$, by optimising the empirical L2-loss: $f(x) = argmin_f \; \sum_i ...
0
votes
0answers
29 views

Incorporating Risk Aversion in Bayesian Expected Loss functions

In Berger's Statistical Decision Theory and Bayesian Analysis, he presents the following expected loss function for decision theory: $\rho(\pi^*,a)=\int_\Theta L(\theta,a)d\pi^*(\theta)$ Where ...
4
votes
1answer
144 views

Bayesian Estimation: Bernoulli and Quadratic Loss Function

I am trying to understand a solution to this problem (I am a very beginner in Bayesian statistics) and I am terribly confused so I would appreciate it if someone could explain to me how exactly this ...
0
votes
0answers
43 views

R- Cost function in glm

I need to introduce a cost function in a logistic model (I'm using R). As I saw from this question, we can introduce costs in cv.glm. But I don't know how to introduce it in glm. My cost function ...
4
votes
0answers
53 views

Baseline for Precision-Related Metrics

When working with ROC-AUC as a metric for binary classification, one often considers a value of 0.5 as a baseline from a random classifier (i.e. a data-blind classifier that randomly classifies test ...
0
votes
0answers
64 views

cost matrix, unbalanced class, oversampling and threshold probability

Let's suppose I have a cost matrix with TP=+90 FP=-10 TN=0 and FN=-10, and that the class is unbalanced. I need to capture the costs in my decision. To do so, I always consider the probability ...
0
votes
0answers
116 views

Loss Functions and Evaluation Metrics

Do you have to evaluate with the same (or equivalent) loss function for model selection purpose? Say you have bunch of models to select. A loss function of one model in training stage is different ...
0
votes
0answers
60 views

How to do optimization using asymmetrical loss functions (LINEX)

Long time Lurker, first time asking. For a research paper, i'm required to optimize some parameters of a certain function using an asymmetrical loss function, specifically LINEX and compare it to the ...
0
votes
1answer
133 views

what does a negative logloss value indicate

I am using logloss python function provided here and I am getting results as -2.99 when I use a machine learning algorithm on my dataset. What does that mean? The algorithm's predictions are bad (or) ...
0
votes
0answers
19 views

Would knowing the underlying distribution for our data affect how boosting searches for its predictor or how it minimizes the exponential loss?

Assume that the goal of Machine learning is to find a function that is able to minimize the generalization/expected/true error (assuming that the underlying distribution is fixed but unknown): $$E(f) ...
0
votes
1answer
27 views

Why is it necessary to assume that examples and labels are drawn from a joint distribution in empirical loss minimization?

Multiple sources have indicated that when trying to minimize empirical loss, $1/N \sum_i L(f(x_i, w), y_i)$, where $L$ is some loss function, $y_i$ is the true label, and $f(x_i, w)$ is the predicted ...
0
votes
0answers
30 views

How to define my custom cost function to be used in (stochastic) gradient descent?

I have a text classification problem were the classes are 20 cities and the input is text Bag of word features. I am using Logistic Regression and my cost function is negative log likelihood: ...
1
vote
0answers
21 views

Is there a version of Buhlmann-Straub credibility that uses an non-fixed $\theta_i$?

Everything I've read about Buhlmann-Straub credibility assumes a fixed $\theta_i$ (the unknown parameter that $X_{ij}$, the variable of interest, depends on). Does anyone know of a version where theta ...
0
votes
1answer
775 views

Multi-class logarithmic loss function per class

In a multi-classification problem, we define the logarithmic loss function $F$ in terms of the logarithmic loss function per label $F_i$ as: $$ F = -\frac{1}{N}\sum_{i}^{N}\sum_{j}^{M}y_{ij} \cdot ...
0
votes
0answers
17 views

In-sample likelihood ratio test for losses

Probably a very simple question for practitioners but I am doing this the very first time: I have written a programm for the estimation of a conditional variance model (HEAVY) of return data and an ...
6
votes
1answer
123 views

Comparing residuals between OLS and non-OLS regressions

Suppose you want to estimate a linear model: ($n$ observations of the response, and $p+1$ predictors) $$\mathbb{E}(y_i) = \beta_0 + \sum_{j=1}^p \beta_j x_{ij}$$ One way to do this is through the OLS ...
6
votes
0answers
214 views

Choosing between loss functions for binary classification

I work in a problem domain where people often report ROC-AUC or AveP (average precision). However, I recently found papers that optimize Log Loss instead, while yet others report Hinge Loss. While I ...
1
vote
0answers
83 views

MHT: Pre-Selecting statistical tests without Bias

Summary: The last formula boxed in red (which is a modified log likelihood from logistic regression) is a special non-differentiable loss function that is adapted to contain a Bonferroni correction in ...
1
vote
0answers
43 views

Squared Error vs Absolute Error loss functions [duplicate]

The two most popular types of loss functions are 1) squared error: $L(y,f(x))=(y-f(x))^2$ --> best estimate is the $E(Y|x) $ 2) absolute error: $L(y,f(x))=|y-f(x)|$ --> best estimate is the ...
3
votes
2answers
781 views

Cost function of neural network is non-convex?

The cost function of neural network is $J(W,b)$, and it is claimed to be non-convex. I don't quite understand why it's that way, since as I see that it's quite similar to the cost function of logistic ...
2
votes
1answer
54 views

regularized loss function

I'm fitting a classifier with cross-entropy loss (i.e. Bernoulli likelihood). Some examples are very clearly associated with one class or the other, and despite some attempts at regularization, the ...
4
votes
1answer
78 views

Loss function that relates ROPE with HDI?

In Doing Bayesian Data Analysis (link to the book) and Bayesian Estimation Supersedes the t-Test, J. Kruschke proposes using the following criterion to reject or accept the null hypothesis in a ...
9
votes
2answers
273 views

Percentile Loss Functions

The solution to the problem: $$ \min_{m} \; E[|m-X|] $$ is well known to be the median of $X$, but what does the loss function look like for other percentiles? Ex: the 25th percentile of X is the ...
1
vote
0answers
52 views

Wilcoxon-Mann-Whitney as a loss function

I'm reading a paper where the authors are using Wilcoxon-Mann-Whitney loss function while minimizing an objective function. As the authors say in the paper, the role of the loss function is to give a ...
0
votes
1answer
225 views

Liblinear types of solver

There is many variants of type of solver in liblinear but I don't understand their differences.Which one I must choose? Also why data must be scaled? duo to some numerical issues? ...
2
votes
1answer
218 views

Interpretation of logloss value

Does anyone have a interpretation of a logloss value? Am I correct to assume that values closer to 0 and 1 are more likely to be an indication that the predicted value is incorrect? Thanks.
0
votes
1answer
61 views

Forcing a particular false positives rate in a learning algorithm

I have a learning algorithm that classifies points as 0 or 1 (haven't settled on which one to implement yet). Of the points I classify as 1, I want to ensure that the number of points correctly ...
4
votes
1answer
309 views

Comparable traing and test cross-entropies result in very different accuracies

Premises I'm training a convolutional neural network (ConvNet) on 51 subclasses in the ImageNet dataset. In order to keep an eye on overfitting, I have been suggested to plot training and testing ...