# Error distributions and loss functions

If I'm right, using the Gaussian distribution in the maximum likelihood estimate yields the mean squared loss.

1. Are there similar relationships between other distributions and losses (say Bernoulli and logistic loss)?
2. How do you derive such relationships mathematically? Note: the context is regression/classification kind of problems where the goal is to predict $y|x$

Let $X_1, X_2, ..., X_n$ be i.i.d with density $ke^{-g(x-\theta)}$

Then $-\log(\mathcal{L}(\theta))= c + \sum_i g(x_i-\theta)$

This gives a big clue about the correspondence between density and what loss function is being minimized with ML.

So for example, if $g$ is $(x-\theta)^2$, we have a least-squares problem and the MLE of $\theta$ will be the sample mean.

If $g$ is $|x-\theta|$ we'll maximize likelihood by minimizing the sum of absolute deviations ... which will give us the median.

--

In response to the question in comments, note that $f=ke^{-|\epsilon|}$ is a Laplace distribution, so the regression model you mention there would correspond to a no-intercept model with conditionally Laplace errors.

• So here g = $Pr(y_{i}|x_{i})$? – wabbit Mar 4 '16 at 8:38
• Yes. Let's say I'm trying to predict Y using $x_{i}s.$So for the Gaussian distribution here g = $Pr(y_{i}|x_{i})=c_{1}e^ {-(y_{i}-\theta x_{i})^2}$. Then what distribution corresponds to $Pr(y_{i}|x_{i}=c_{1}e^{-|y_{i}-\theta x_{i}|}$ – wabbit Mar 4 '16 at 10:08
• @Wabbit thanks for the edit. Note that your errors $y_i-\theta x_i$ contain no intercept term (when $x=0$ then under symmetric losses you'd try to put your fitted $y$ near $0$). Was that deliberate or did you mean to allow lines to have their intercept somewhere other than 0? – Glen_b -Reinstate Monica Mar 6 '16 at 0:31