A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

learn more… | top users | synonyms

0
votes
0answers
14 views

Changing optimization algorithm while optimizing

I am currently training some convolutional neural networks with cross-entropy loss. Thus, the function I am optimizing is non-convex, and at the moment I am using an optimization algorithm called Adam ...
1
vote
2answers
56 views

How can I force my model to predict the samples that are close to zero?

I have a large amount of inventory data and I am trying to predict when the inventory gets low using one component of the change in inventory (yes I know this doesn't describe inventory very well by ...
0
votes
0answers
15 views

derive loss function for gamma regression

In the R package mboost there is a family called "GammaReg" which implementes "negative Gamma log-likelihood with logarithmic link function". Still, I don't really ...
3
votes
1answer
53 views

Shrinkage of the Sample Covariance matrix

Assume we have N independent and identically distributed random vectors $X_1, X_2, ..., X_N$ where each of them is of size p $\times$ 1. The sample covariance matrix, denoted here by $S$, is computed ...
3
votes
1answer
75 views

What's the measure to assess the binary classification accuracy for imbalanced data?

Now I have binary classification problem with positive samples roughly 100 times the number of negative samples. In this case the normal accuracy measure (predict == label) is not a good measure. What ...
0
votes
1answer
33 views

Why linear regression one variable uses squared cost function? [duplicate]

This is about linear regression course given by Andrew Ng on Coursea about machine learning. why cost function is $$ \frac{1}{m} \sum _{i=1}^m \left(h_\theta(X^{(i)})-Y^{(i)}\right)^2 $$ and not ...
1
vote
1answer
35 views

derivative of loss function

If we have a paralyzed loss function of the form of: \begin{align} L(\beta)& =\frac{1}{2}(y-X\beta)^T(y-X\beta)+ \lambda \beta^T f(\beta) \end{align} where $X_{n\times m}$ and $\beta_{m \times ...
2
votes
1answer
105 views

Scikit Binomial Deviance Loss Function

This is scikit GradientBoosting's binomial deviance loss function, ...
1
vote
3answers
172 views

Best loss function for very sparse real-valued data

Suppose the target output of my data prediction model is an $M\times N$ matrix where $95\%$ of the values are $0.0$ and the other values are anywhere between $0.0$ and $1.0$, what would be a good loss ...
0
votes
0answers
53 views

Gradient for hinge loss multiclass

I am little confused when trying to find the gradient for the multiclass hinge loss: $l(y) = \max( 0, 1 + \underset{r \neq y_i}{ \text{max} \ } W_r \cdot x_i + W_{y_i} \cdot x_i)$ Where $W^{k \times ...
1
vote
0answers
109 views

Loss and dropout in deep learning

I have a CNN with 3 convolutional layers, 1 max-pooling layer and 2 fully-connected layers before applying softmax classification. The CNN is trained with Adagrad and I achieve a quite good ...
0
votes
0answers
22 views

Logistic loss approximation

In many implementations of logistic loss (example sklearn) I see the following code(adapted from sklearn), where p is the prediction and y the true value: ...
1
vote
1answer
61 views

Gradient of loss function for (non)-linear prediction functions

$ \newcommand{\y}{\mathbf{y}} \newcommand{\wv}{\mathbf{w}} \newcommand{\xv}{\mathbf{x}} \newcommand{\loss}{L(\wv;\xv, y)} $ I'm trying to clear up the calculation of the gradient of a loss function, ...
0
votes
0answers
47 views

Meaning of the prior and loss parameters in rpart in R

Could someone please explain to me what specifying priors and/or loss parameters in R's rpart actually do? I found R's documentation completely unhelpful. For example, let's suppose I have a highly ...
1
vote
0answers
47 views

hinge loss vs logistic loss advantages and disadvantages/limitations

Hinge loss can be defined using $\text{max}(0, 1-y_i\mathbf{w}^T\mathbf{x}_i)$ and the log loss can be defined as $\text{log}(1 + \exp(-y_i\mathbf{w}^T\mathbf{x}_i))$ I have the following questions: ...
0
votes
0answers
30 views

maximum mis-classification loss of hinge loss

In the plots and in some lecture notes, I read that hinge loss is bounded between (0,2). But I can not understand that. By definition, hinge loss is (standard one) ...
4
votes
1answer
65 views

showing that $\bar{X}$ is inadmissible by comparing with $\max(\bar{X},2)$ under squared error loss function

suppose $X_1,X_2,\ldots,X_n$ be a random sample of $N(\theta,1), \theta>2$. how can I show $\bar{X}$ is inadmissible estimator Compared to $\max(\bar{X},2)$ under Squared error loss function
2
votes
0answers
35 views

Does maximum likelihood minimize a kind of generalized “0-1 loss”?

A very good point was raised here about how the optimal betting strategy under 0-1 loss was to bet on the mode, while under MSE loss the optimal strategy was to bet on the mean. Maximum likelihood ...
0
votes
3answers
57 views

Equating the two equations of Ridge Regression

I'm studying Ridge Regression now and I'm having a bit of trouble understanding how to relate the two equations that pop up when I read about it. There is the coefficient estimate: $$\hat{\beta} = ...
7
votes
1answer
116 views

What would be an example of when L2 is a good loss function for computing a posterior loss?

L2 loss, together with L0 and L1 loss, are three a very common "default" loss functions used when summarising a posterior by the minimum posterior expected loss. One reason for this is perhaps that ...
0
votes
0answers
133 views

Implementing WARP Loss (Gradient Computation)

I am trying to implement the WARP Loss in Torch, as defined in the WSABIE paper: http://www.thespermwhale.com/jaseweston/papers/wsabie-ijcai.pdf The Algorithm is as follows: The Algorithm ...
0
votes
0answers
72 views

Value of the loss function and Classification Errors in gbm package (R)

I have a simple problem of classification (0s and 1s) using adaboost loss function. When I check the components of a boosted model using the gbm package I see: ...
1
vote
0answers
35 views

Loss function from Posterior log likelihood

I am trying to implement Bayesian Logistic Regression and was looking at this paper http://www.stat.columbia.edu/~madigan/PAPERS/techno.pdf. The authors in this paper had assumed the $\beta$s to have ...
1
vote
0answers
29 views

Loss function for rank deficient covariance matrices?

I'm trying to compare the efficiency of different estimators of the covariance matrix of a particular type of multivariate normally distributed data. This comparison, as well as the estimation process ...
2
votes
1answer
123 views

Get distribution for aggregate loss using Monte Carlo

I am given two data sets containing dates and losses (in some currency). Given a distribution for the amount of losses and an (a,b,0) distribution for frequency of losses, how can I use Monte Carlo ...
3
votes
1answer
79 views

Designing Asymmetric regression (assymettric loss for regression)

I have a hybrid classification/regression problem.The predicted value can be assumed to be centred around 0. I want to penalize the predictor more, if the predicted value and actual value have ...
2
votes
1answer
116 views

minimax property of sample mean

Suppose $X_1,X_2,\ldots,X_n$ are iid $\mathcal{N}(\mu,\sigma^2)$, where $\sigma$ is known, but $\mu$ is not. We wish to construct a confidence interval of length $L$ (given) for $\mu$. Is it true that ...
0
votes
1answer
47 views

Estimating conditional variance y|x

I am building a predictor for $y = f(x)$ using training samples ${(x_i, y_i)}$ (assume) drawn i.i.d from some distribution $p(x,y)$, by optimising the empirical L2-loss: $f(x) = argmin_f \; \sum_i ...
0
votes
0answers
29 views

Incorporating Risk Aversion in Bayesian Expected Loss functions

In Berger's Statistical Decision Theory and Bayesian Analysis, he presents the following expected loss function for decision theory: $\rho(\pi^*,a)=\int_\Theta L(\theta,a)d\pi^*(\theta)$ Where ...
4
votes
1answer
171 views

Bayesian Estimation: Bernoulli and Quadratic Loss Function

I am trying to understand a solution to this problem (I am a very beginner in Bayesian statistics) and I am terribly confused so I would appreciate it if someone could explain to me how exactly this ...
0
votes
0answers
55 views

R- Cost function in glm

I need to introduce a cost function in a logistic model (I'm using R). As I saw from this question, we can introduce costs in cv.glm. But I don't know how to introduce it in glm. My cost function ...
4
votes
0answers
61 views

Baseline for Precision-Related Metrics

When working with ROC-AUC as a metric for binary classification, one often considers a value of 0.5 as a baseline from a random classifier (i.e. a data-blind classifier that randomly classifies test ...
0
votes
0answers
75 views

cost matrix, unbalanced class, oversampling and threshold probability

Let's suppose I have a cost matrix with TP=+90 FP=-10 TN=0 and FN=-10, and that the class is unbalanced. I need to capture the costs in my decision. To do so, I always consider the probability ...
0
votes
0answers
147 views

Loss Functions and Evaluation Metrics

Do you have to evaluate with the same (or equivalent) loss function for model selection purpose? Say you have bunch of models to select. A loss function of one model in training stage is different ...
0
votes
0answers
64 views

How to do optimization using asymmetrical loss functions (LINEX)

Long time Lurker, first time asking. For a research paper, i'm required to optimize some parameters of a certain function using an asymmetrical loss function, specifically LINEX and compare it to the ...
0
votes
1answer
145 views

what does a negative logloss value indicate

I am using logloss python function provided here and I am getting results as -2.99 when I use a machine learning algorithm on my dataset. What does that mean? The algorithm's predictions are bad (or) ...
0
votes
0answers
19 views

Would knowing the underlying distribution for our data affect how boosting searches for its predictor or how it minimizes the exponential loss?

Assume that the goal of Machine learning is to find a function that is able to minimize the generalization/expected/true error (assuming that the underlying distribution is fixed but unknown): $$E(f) ...
0
votes
1answer
27 views

Why is it necessary to assume that examples and labels are drawn from a joint distribution in empirical loss minimization?

Multiple sources have indicated that when trying to minimize empirical loss, $1/N \sum_i L(f(x_i, w), y_i)$, where $L$ is some loss function, $y_i$ is the true label, and $f(x_i, w)$ is the predicted ...
0
votes
0answers
45 views

How to define my custom cost function to be used in (stochastic) gradient descent?

I have a text classification problem were the classes are 20 cities and the input is text Bag of word features. I am using Logistic Regression and my cost function is negative log likelihood: ...
1
vote
0answers
22 views

Is there a version of Buhlmann-Straub credibility that uses an non-fixed $\theta_i$?

Everything I've read about Buhlmann-Straub credibility assumes a fixed $\theta_i$ (the unknown parameter that $X_{ij}$, the variable of interest, depends on). Does anyone know of a version where theta ...
0
votes
1answer
884 views

Multi-class logarithmic loss function per class

In a multi-classification problem, we define the logarithmic loss function $F$ in terms of the logarithmic loss function per label $F_i$ as: $$ F = -\frac{1}{N}\sum_{i}^{N}\sum_{j}^{M}y_{ij} \cdot ...
6
votes
1answer
133 views

Comparing residuals between OLS and non-OLS regressions

Suppose you want to estimate a linear model: ($n$ observations of the response, and $p+1$ predictors) $$\mathbb{E}(y_i) = \beta_0 + \sum_{j=1}^p \beta_j x_{ij}$$ One way to do this is through the OLS ...
6
votes
0answers
255 views

Choosing between loss functions for binary classification

I work in a problem domain where people often report ROC-AUC or AveP (average precision). However, I recently found papers that optimize Log Loss instead, while yet others report Hinge Loss. While I ...
1
vote
0answers
87 views

MHT: Pre-Selecting statistical tests without Bias

Summary: The last formula boxed in red (which is a modified log likelihood from logistic regression) is a special non-differentiable loss function that is adapted to contain a Bonferroni correction in ...
1
vote
0answers
44 views

Squared Error vs Absolute Error loss functions [duplicate]

The two most popular types of loss functions are 1) squared error: $L(y,f(x))=(y-f(x))^2$ --> best estimate is the $E(Y|x) $ 2) absolute error: $L(y,f(x))=|y-f(x)|$ --> best estimate is the ...
4
votes
2answers
974 views

Cost function of neural network is non-convex?

The cost function of neural network is $J(W,b)$, and it is claimed to be non-convex. I don't quite understand why it's that way, since as I see that it's quite similar to the cost function of logistic ...
2
votes
1answer
57 views

regularized loss function

I'm fitting a classifier with cross-entropy loss (i.e. Bernoulli likelihood). Some examples are very clearly associated with one class or the other, and despite some attempts at regularization, the ...
4
votes
1answer
85 views

Loss function that relates ROPE with HDI?

In Doing Bayesian Data Analysis (link to the book) and Bayesian Estimation Supersedes the t-Test, J. Kruschke proposes using the following criterion to reject or accept the null hypothesis in a ...
9
votes
2answers
278 views

Percentile Loss Functions

The solution to the problem: $$ \min_{m} \; E[|m-X|] $$ is well known to be the median of $X$, but what does the loss function look like for other percentiles? Ex: the 25th percentile of X is the ...
1
vote
0answers
56 views

Wilcoxon-Mann-Whitney as a loss function

I'm reading a paper where the authors are using Wilcoxon-Mann-Whitney loss function while minimizing an objective function. As the authors say in the paper, the role of the loss function is to give a ...