A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

learn more… | top users | synonyms

0
votes
0answers
2 views

How to define my custom cost function to be used in (stochastic) gradient descent?

I have a text classification problem were the classes are 20 cities and the input is text Bag of word features. I am using Logistic Regression and my cost function is negative log likelihood: ...
1
vote
0answers
18 views

Is there a version of Buhlmann-Straub credibility that uses an non-fixed $\theta_i$?

Everything I've read about Buhlmann-Straub credibility assumes a fixed $\theta_i$ (the unknown parameter that $X_{ij}$, the variable of interest, depends on). Does anyone know of a version where theta ...
0
votes
0answers
18 views

Multi-class logarithmic loss function per class

In a multi-classification problem, we define the logarithmic loss function $F$ in terms of the logarithmic loss function per label $F_i$ as: $$ F = -\frac{1}{N}\sum_{i}^{N}\sum_{j}^{M}y_{ij} \cdot ...
0
votes
0answers
9 views

In-sample likelihood ratio test for losses

Probably a very simple question for practitioners but I am doing this the very first time: I have written a programm for the estimation of a conditional variance model (HEAVY) of return data and an ...
6
votes
1answer
83 views

Comparing residuals between OLS and non-OLS regressions

Suppose you want to estimate a linear model: ($n$ observations of the response, and $p+1$ predictors) $$\mathbb{E}(y_i) = \beta_0 + \sum_{j=1}^p \beta_j x_{ij}$$ One way to do this is through the OLS ...
3
votes
0answers
25 views

Choosing between loss functions for binary classification

I work in a problem domain where people often report ROC-AUC or AveP (average precision). However, I recently found papers that optimize Log Loss instead, while yet others report Hinge Loss. While I ...
1
vote
0answers
70 views

MHT: Pre-Selecting statistical tests without Bias

Summary: The last formula boxed in red (which is a modified log likelihood from logistic regression) is a special non-differentiable loss function that is adapted to contain a Bonferroni correction in ...
1
vote
0answers
36 views

Squared Error vs Absolute Error loss functions [duplicate]

The two most popular types of loss functions are 1) squared error: $L(y,f(x))=(y-f(x))^2$ --> best estimate is the $E(Y|x) $ 2) absolute error: $L(y,f(x))=|y-f(x)|$ --> best estimate is the ...
0
votes
1answer
67 views

Cost function of neural network is NON-CONVEX?

The cost function of neural network is $J(W,b)$, and it is claimed to be non-convex. I don't quite understand why it's that way, since as I see that it's quite similar to the cost function of Logistic ...
2
votes
1answer
39 views

regularized loss function

I'm fitting a classifier with cross-entropy loss (i.e. Bernoulli likelihood). Some examples are very clearly associated with one class or the other, and despite some attempts at regularization, the ...
3
votes
1answer
50 views

Loss function that relates ROPE with HDI?

In Doing Bayesian Data Analysis (link to the book) and Bayesian Estimation Supersedes the t-Test, J. Kruschke proposes using the following criterion to reject or accept the null hypothesis in a ...
8
votes
2answers
207 views

Percentile Loss Functions

The solution to the problem: $$ \min_{m} \; E[|m-X|] $$ is well known to be the median of $X$, but what does the loss function look like for other percentiles? Ex: the 25th percentile of X is the ...
0
votes
0answers
17 views

How to predict extremes of a continuous response variable?

Suppose I have a response $Y_t \sim N(0,\sigma^2_Y)$ and features $X_{i,t} \sim N(0,\sigma^2_{X_i})$ for $i \in \{1,...,100\}$. $Y_t$ conforms to a linear model $Y_t = a + \sum_{i=1}^{100} b_i ...
1
vote
0answers
38 views

Wilcoxon-Mann-Whitney as a loss function

I'm reading a paper where the authors are using Wilcoxon-Mann-Whitney loss function while minimizing an objective function. As the authors say in the paper, the role of the loss function is to give a ...
0
votes
0answers
40 views

Liblinear types of solver

There is many variants of type of solver in liblinear but I don't understand their differences.Which one I must choose? Also why data must be scaled? duo to some numerical issues? ...
2
votes
1answer
36 views

Interpretation of logloss value

Does anyone have a interpretation of a logloss value? Am I correct to assume that values closer to 0 and 1 are more likely to be an indication that the predicted value is incorrect? Thanks.
0
votes
1answer
48 views

Forcing a particular false positives rate in a learning algorithm

I have a learning algorithm that classifies points as 0 or 1 (haven't settled on which one to implement yet). Of the points I classify as 1, I want to ensure that the number of points correctly ...
4
votes
1answer
149 views

Comparable traing and test cross-entropies result in very different accuracies

Premises I'm training a convolutional neural network (ConvNet) on 51 subclasses in the ImageNet dataset. In order to keep an eye on overfitting, I have been suggested to plot training and testing ...
2
votes
0answers
38 views

What is a robust way to find the max of $n$ independent, non-identical random variates?

Suppose I observe $n$ random variates along with their variance (but not mean) and I'd like to select the one with the largest mean as frequently as possible. The procedure must be memoryless--you ...
4
votes
1answer
49 views

Relation between Scoring rule and Loss function in Parameter estimation and model selection?

Initially, I had only heard of MLE and use it for almost everything, e.g. point estimate and model selection (with some penalty). Then, MSE appeared, which seems to play the same role as MLE does. I ...
3
votes
1answer
53 views

Question about computing Bayes Error - with or without loss function?

I am new to Bayesian Decision Theory and don't understand the following concept: So from what I understood, the Bayes error is used to report the performance of a Bayes classifier in terms of the ...
0
votes
1answer
72 views

How to design a cost function that has different weights for different types of classification errors?

I'm trying to design a continuous loss function for a logistic classifier. Suppose I have the following confusion matrix: [tn fp fn tp] I want the loss ...
0
votes
0answers
39 views

Properties of the Hellinger distance as a loss function

I am thinking about different loss function for non-parametric estimators (of densities). It is often said that $L^2$ loss is sensitive to outliers and doesn't do a good job of representing the tail ...
2
votes
3answers
188 views

Importance of optimizing the correct loss function

I want to understand the importance of optimizing the correct loss function. Say that I am building a linear regression model $p$ for predicting some values $y_1,\ldots,y_n$. I choose to fit my ...
0
votes
0answers
14 views

Loss specific belief propagation

We know standard Belief Propagation finds the parameters which maximize probability of posterior. Is there any way to use BP, for loss specific inference? For example, let's say someone wants to ...
0
votes
0answers
45 views

Maximizing F1-measure, when you have an algorithm for minimizing another loss function

Let's say you have an algorithm for minimizing the following loss function: $$ loss = \sum_i l(y_i, f(x_i)) $$ Let's say you are in the binary classification case, and the ratio of negative to ...
2
votes
1answer
144 views

Minimalizing absolute cost function error

I got two questions. 1. I know that in predictive analytics contests, when faced to yes/no problems, with the absolute cost function $f(x) = \frac{1}{n}\sum_{i=1}^n\lvert x_i-\hat x_i\rvert$ the ...
3
votes
1answer
213 views

LASSO with L1 loss function

I've been trying to figure out a way to perform LASSO with L1 loss function (instead of the L2 loss) but have been completely dumfounded as to how. I've attempted to use the flare package's ...
1
vote
1answer
52 views

Is the F-1 score symmetric?

Below is the report of my out-of-bag precision, recall and f-1score when using ...
0
votes
0answers
68 views

Loss function of kernlab

I'm looking for the default loss function the ksvm()-function from R package 'kernlab' is using. My guess is the Hinge Loss but I cant find any reference or citation for it (There's nothing ...
4
votes
1answer
166 views

What is the loss function of hard margin SVM?

People says soft margin SVM use hinge loss function: $\max(0,1-y_i(w^\intercal x_i+b))$. However, the actual objective function that soft margin SVM tries to minimize is $$ ...
0
votes
0answers
59 views

Comparing different types of losses as functions of lambda?

The usual pictures we see when dealing with different loss functions look similar to this: Here we see y*f(x) on the x-axis with an error associated with it. Suppose I have a logistic regression ...
0
votes
1answer
131 views

What is the loss function for C - Support Vector Classification?

In article LIBSVM: A Library for Support Vector Machines there is written, than C-SVC uses loss function: $$ \frac{1}{2}w^Tw+C\sum\limits_{i=1}^l\xi_i$$ OK, I know, what is $w^Tw$. But what is ...
0
votes
0answers
48 views

Bayesian Expected Loss for mean F1-score loss function

So I have a multi-label classification problem where the exact number of labels in each test set example is unknown. The loss function is mean F1-score which is where p is the precision and r is ...
2
votes
0answers
73 views

Regression analysis with special cost function

I want to do regression analysis using a special cost function that penalizes the sign error more than the square of the error. For example, I have a number of monthly change observations that can ...
1
vote
0answers
130 views

Why isn't k-means optimized using gradient descent?

I know k-means is usually optimized using Expectation Maximization. However we could optimize its loss function the same way we optimize any other! I found some papers that actually use stochastic ...
2
votes
0answers
19 views

Risk bounds for Poisson regression

Defining risk of the estimator as the expected square loss, is there any work discussing the non-asymptotic risk bounds for Poisson regression based on maximum likelihood estimate (MLE)?
2
votes
0answers
54 views

Transforming an estimator to overestimate

I have an estimator $\mu^*$ of a the mean $\mu$ of a certain distribution that I obtained using a variational technique (basically just establishing a bound on $\mu$ and finding a trial function that ...
1
vote
1answer
71 views

SVM with quadratic loss

I've seen some statement where I got the impression that SVM with a quadratic loss is no more than having a kernel matrix where a multiple of the unit matrix is subtracted from the kernel. It was ...
0
votes
2answers
181 views

Minimizing the expected loss

I was wondering about the motivation behind the following definition of expected loss: $$E[L] = \sum_{k} \sum_{j} \int_{R_{j}} L_{kj} p(x, C_{k})dx$$ where $L_{kj}$ is the loss matrix, in which $j$ ...
3
votes
2answers
207 views

Minimizing variance of an estimator under sampling cost penalty

I have an estimator $t$, whose variance depends on the dimension of my sample $x_{1:n}$: $$ \text{Var}(t(x_{1:n})) = f(n). $$ Suppose that the form of $f(n)$ is known. I would like to determine what ...
2
votes
0answers
89 views

A “Gambler's Loss function”?

What is a good loss function for a predictive model used by gamblers? I've been reading a bit about loss functions recently. I've always just went with MSE (e.g., for a couple of neural network ...
4
votes
1answer
129 views

Expectation notations

In Statistical Decision Theory, one often studies the following two measures (from "The Bayesian Choice"): Average loss (aka the frequentist risk): $R\left(\theta,\delta\right) = ...
1
vote
0answers
62 views

Can I define an $R^2$-like measure in this way when predicting with exotic loss functions?

It is common in regression to see $R^2$ formulated as follows: $$R^2\equiv 1 - {SS_{\rm err}\over SS_{\rm tot}},$$ where $SS_\text{err}=\sum_i (y_i - f_i)^2$ and $SS_\text{tot}=\sum_i ...
2
votes
1answer
147 views

Notation in GBM package vignette: expected value of loss functions

Can anyone help with the understanding of this notation (and idea) from the vignette for GBM in R? It starts with the following: Question 1: I believe this is simply saying that we are looking for ...
11
votes
2answers
555 views

How to design and implement an asymmetric loss function for regression?

Problem In regression one usually computes the mean squared error (MSE) for a sample: $$ \text{MSE} = \frac{1}{n} \sum_{i=1}^n\left(g(x_i) - \widehat{g}(x_i)\right)^2 $$ to measure the quality of a ...
6
votes
1answer
978 views

L1 regression estimates median whereas L2 regression estimates mean?

So I was asked a question on which central measures L1 (i.e., lasso) and L2 (i.e., ridge regression) estimated. The answer is L1=median and L2=mean. Is there any type of intuitive reasoning to this? ...
2
votes
0answers
107 views

Asymptotics of 0-1 classification loss

I am interested in training a simple binary linear classifier. That is, I will find a vector of weights $\bf w$ such that I can predict the class of new example by the sign of $f(x) = w^T x$. I ...
2
votes
0answers
166 views

Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$ r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]], $$ where $L$ is some loss function and $\hat x$ is the ...
2
votes
1answer
74 views

Convexity of loss function with respect to the mean

Let $X\,$ be a non-negative r.v. with known pdf $f(x|\theta)$ but with a single unknown parameter $\theta$. Suppose that the mean $\mu$ can be used to uniquely determine the value of $\theta$, i.e. if ...