5
$\begingroup$

Simple question: Can we generally think of the loss function as the negative of the likelihood function?

For instance with regards to logistic regression, the likelihood function in a binary setting is

$\sum_i y^{i}\log(h(x^i))+\log(1-y^i)(1-h(x^i))$

while the loss function is

$- \Big[\sum_i y^{i}\log(h(x^i))+\log(1-y^i)(1-h(x^i))\Big]$

However, in Maximum-A-Posteriori (MAP) tasks I have seen that the loss function is derived by maximizing the posterior, i.e. the loss function being the differentiation of the likelihood function times the prior.

$\endgroup$
  • $\begingroup$ The first equation is the log likelihood. $\endgroup$ – user0 Aug 2 '17 at 14:52
3
$\begingroup$

In linear regression and logistic regression, without regularization, we can think the objective is to maximize likelihood.

On the other hand, we the term "loss function" is more general than likelihood.

For example, we can add regularization (See Regularization methods for logistic regression).

We can also add other constraints or use weighted loss. All of these are "add on" to likelihood.

Check this post for details

Objective function, cost function, loss function: are they the same thing?

$\endgroup$
  • $\begingroup$ What I mean is this: When we optimize with MAP, we maximize the posterior, i.e. we maximize likelihood x prior. In the logisitic regression context, we maximize however the likelihood function as quoted above. This likelihood function however has class conditioned on data similar to the posterior of say a bayesian classifier. Sorry if it is not understandable, not sure how to put it differently $\endgroup$ – Pegah Aug 3 '17 at 15:57
  • 1
    $\begingroup$ @Pegah MAP can be viewed as a generative model and logistic regression is a discriminative model, which we only focusing on getting $P(Y|X)$. this paper may help. ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf $\endgroup$ – Haitao Du Aug 3 '17 at 16:00
  • $\begingroup$ I agree with the things you said in your answer. Regarding the last comment, I wouldn't say that MAP is a generative model. It's an approach for estimating parameters (not a model at all), and can be used with either generative or discriminative models. For example, MAP estimation could be used to fit a logistic regression model w/ priors, which is a discriminative model. $\endgroup$ – user20160 Aug 5 '17 at 4:27
  • $\begingroup$ Also, linear regression is only equivalent to MLE when we assume the error has a normal distribution. $\endgroup$ – James May 10 '18 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.