Gaussian prior in Bayesian setting is equivalent to minimizing squared error, while Laplace prior minimizes the absolute error and leads to lasso regression. What (if any) prior distribution can be thought as an alternative to Huber loss function?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Let's see what happens when we consider the likelihood.
In each case you mention, your density is proportional to $e^{-k.L(x-\mu)}$.
We can see that this will be the case more generally than just those cases, because taking this density and writing the negative log-likelihood we get $\sum_i L(x_i-\mu)$ back as an object to minimize.
Consequently we get a density with a Gaussian center and exponential tails yield a Huber loss.
This is sometimes called the Huber density.
See $\rho$ functions in M-estimators, where the connection between density(/likelihood) and loss function is often made explicit. (This is the context in which Huber developed the loss function.)
-
$\begingroup$ Thanks, but could you extend the "each case you mention" part, because I am not 100% sure if I follow what you wanted to say..? $\endgroup$– TimCommented Nov 10, 2015 at 14:02
-
2$\begingroup$ @Tim Take $L = \frac12 (\theta-x)^2 $. The Gaussian density is $\propto \exp(-L)$. Take $L = |\theta-x|$. The Laplace density is $\propto \exp(-L)$. My exposition is a little handwavy because you haven't made it clear exactly what loss calculation you're performing here (this is why I moved to talking about the likelihood). If you show a particular progression from $L$ to the densities you mention or vice versa (which amounts to being explicit about setting up the problem fully in some specific way that they do correspond), it should be possible to do the same thing here. $\endgroup$– Glen_bCommented Nov 10, 2015 at 14:05