10
$\begingroup$

Simple question: Can we generally think of the loss function as the negative of the likelihood function?

For instance with regards to logistic regression, the likelihood function in a binary setting is

$\sum_i y^{i}\log(h(x^i))+\log(1-y^i)(1-h(x^i))$

while the loss function is

$- \Big[\sum_i y^{i}\log(h(x^i))+\log(1-y^i)(1-h(x^i))\Big]$

However, in Maximum-A-Posteriori (MAP) tasks I have seen that the loss function is derived by maximizing the posterior, i.e. the loss function being the differentiation of the likelihood function times the prior.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ The first equation is the log likelihood. $\endgroup$
    – Sam Weisenthal
    Commented Aug 2, 2017 at 14:52

1 Answer 1

Reset to default
7
$\begingroup$

In linear regression and logistic regression, without regularization, we can think the objective is to maximize likelihood.

On the other hand, we the term "loss function" is more general than likelihood.

For example, we can add regularization (See Regularization methods for logistic regression).

We can also add other constraints or use weighted loss. All of these are "add on" to likelihood.

Check this post for details

Objective function, cost function, loss function: are they the same thing?

Improve this answer
$\endgroup$
4
  • $\begingroup$ What I mean is this: When we optimize with MAP, we maximize the posterior, i.e. we maximize likelihood x prior. In the logisitic regression context, we maximize however the likelihood function as quoted above. This likelihood function however has class conditioned on data similar to the posterior of say a bayesian classifier. Sorry if it is not understandable, not sure how to put it differently $\endgroup$
    – Pugl
    Commented Aug 3, 2017 at 15:57
  • 1
    $\begingroup$ @Pegah MAP can be viewed as a generative model and logistic regression is a discriminative model, which we only focusing on getting $P(Y|X)$. this paper may help. ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf $\endgroup$
    – Haitao Du
    Commented Aug 3, 2017 at 16:00
  • $\begingroup$ I agree with the things you said in your answer. Regarding the last comment, I wouldn't say that MAP is a generative model. It's an approach for estimating parameters (not a model at all), and can be used with either generative or discriminative models. For example, MAP estimation could be used to fit a logistic regression model w/ priors, which is a discriminative model. $\endgroup$
    – user20160
    Commented Aug 5, 2017 at 4:27
  • $\begingroup$ Also, linear regression is only equivalent to MLE when we assume the error has a normal distribution. $\endgroup$
    – James LT
    Commented May 10, 2018 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.