3
$\begingroup$

If I'm right, using the Gaussian distribution in the maximum likelihood estimate yields the mean squared loss.

  1. Are there similar relationships between other distributions and losses (say Bernoulli and logistic loss)?
  2. How do you derive such relationships mathematically? Note: the context is regression/classification kind of problems where the goal is to predict $y|x$
$\endgroup$
5
$\begingroup$

Let $X_1, X_2, ..., X_n$ be i.i.d with density $ke^{-g(x-\theta)}$

Then $-\log(\mathcal{L}(\theta))= c + \sum_i g(x_i-\theta)$

This gives a big clue about the correspondence between density and what loss function is being minimized with ML.

So for example, if $g$ is $(x-\theta)^2$, we have a least-squares problem and the MLE of $\theta$ will be the sample mean.

If $g$ is $|x-\theta|$ we'll maximize likelihood by minimizing the sum of absolute deviations ... which will give us the median.

--

In response to the question in comments, note that $f=ke^{-|\epsilon|}$ is a Laplace distribution, so the regression model you mention there would correspond to a no-intercept model with conditionally Laplace errors.

[I've ignored scale in this answer but it doesn't alter anything substantive about the answer.]

$\endgroup$
  • $\begingroup$ So here g = $Pr(y_{i}|x_{i})$? $\endgroup$ – wabbit Mar 4 '16 at 8:38
  • $\begingroup$ Your question contains nothing whatever about there being two variables in your problem. You have some regression type problem? $\endgroup$ – Glen_b -Reinstate Monica Mar 4 '16 at 9:56
  • $\begingroup$ Yes. Let's say I'm trying to predict Y using $x_{i}s. $So for the Gaussian distribution here g = $Pr(y_{i}|x_{i})=c_{1}e^ {-(y_{i}-\theta x_{i})^2}$. Then what distribution corresponds to $Pr(y_{i}|x_{i}=c_{1}e^{-|y_{i}-\theta x_{i}|}$ $\endgroup$ – wabbit Mar 4 '16 at 10:08
  • $\begingroup$ There is still no indication of this in your question; if that's what you mean to ask about, please edit. It is not clear to me how you thought people would guess this (implied by your first comment under my answer). $\endgroup$ – Glen_b -Reinstate Monica Mar 5 '16 at 0:12
  • $\begingroup$ @Wabbit thanks for the edit. Note that your errors $y_i-\theta x_i$ contain no intercept term (when $x=0$ then under symmetric losses you'd try to put your fitted $y$ near $0$). Was that deliberate or did you mean to allow lines to have their intercept somewhere other than 0? $\endgroup$ – Glen_b -Reinstate Monica Mar 6 '16 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.