I am reading Ian Goodfellow's book about deep learning and when it introduces exponential distribution, it says "In the context of deep learning, we often want to have a probability distribution with a sharp point at x=0." I don't understand why the probability distribution should have this character, does it help to train the model?
6
$\begingroup$
$\endgroup$
-
$\begingroup$ Neither density is differentiable at the cusp point ($0$ for the first, $\mu$ for the second). You can see the sharpness from a plot. Why this is useful in deep learning, I have no idea. I can't really help you there. But also, I take issue with the author's use of the word "mass." $\endgroup$ – Taylor Apr 26 '17 at 3:57
-
$\begingroup$ @Taylor "mass" is standard terminology in probability. $\endgroup$ – jth Apr 26 '17 at 4:38
-
$\begingroup$ @Taylor I think "mass" is a typo. The correct one should be "density" because Laplace distribution is a continuous probability distribution. $\endgroup$ – apepkuss Sep 11 '17 at 15:48
-
1$\begingroup$ cf the relationship between lasso and pinning some coefficient estimates at exactly 0. $\endgroup$ – Sycorax says Reinstate Monica Sep 11 '17 at 15:52
-
1$\begingroup$ In what context is the distribution used? $\endgroup$ – Miguel Oct 30 '17 at 17:47
1
$\begingroup$
$\endgroup$
These two distribution have a connection with deep learning via regularisation. In deep learning we are often concerned with regularising the parameters of a neural network because neural networks tend to overfit and we want to improve ability of the model to generalise to new data.
From a Bayesian perspective, fitting a regularised model can be interpreted as computing the maximum a posteriori (MAP) estimate given a specific prior distribution over the weights $w_i$. In particular,
- the $L^2$ (a.k.a. weight decay) norm corresponds to a Gaussian prior on the weights $w$, and
- the $L^1$ norm corresponds to an isotropic Laplace prior over the weights $w$.
The $L^1$ norm (a.k.a. the Laplace prior on weights) by virtue of it's sharpness encourages sparsity (many zeros) in $w$ for reasons explained here: Why L1 norm for sparse models . This type of regularisation can be quite desirable. The connection between the Laplacian distribution and the $L^1$ norm is explained in more detail here: Why is Lasso penalty equivalent to the double exponential (Laplace) prior?
Most of what I have mentioned here is discussed in more detail in the same "Deep Learning" book in section 5.6.1 and section 7.1.2.
answered Jan 23 '18 at 5:50
