Optimizing Gaussian negative log-likelihood I have been using the PyTorch class torch.nn.GaussianNLLLoss for an optimization problem that I have. It optimizes the mean ($target$) and variance ($var$) of a distribution over a batch $i$ using the formula:
$\text{loss} = \frac{1}{2}\sum_{i=1}^D \left(\log\left(\text{max}\left(\text{var}[i],
        \ \text{eps}\right)\right) + \frac{\left(\text{input}[i] - \text{target}[i]\right)^2}
        {\text{max}\left(\text{var}[i], \ \text{eps}\right)}\right) + \text{const.}$
However, although the loss makes intuitive sense I don't understand where the $\log\left(\text{max}\left(\text{var}[i],
        \ \text{eps}\right)\right)$ term comes from. I would like to modify this code to optimize for the exponentially modified Gaussian distribution but I am unsure how to include the new terms.
 A: The regular Gaussian likelihood of a single value $y$, given parameters $\mu$ and $\sigma$ would be:
$$
N(y;\mu,\sigma)={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}{2}}\left({\frac {y-\mu }{\sigma }}\right)^{2}}
$$
I used $y$ instead of $x$ to avoid confusion later. In order to optimize a neural network one needs it's logarithm. You can use property of the logarithm:
$$
\log(ab) = \log(a) + \log(b)
$$
and separate the normalizing "constant":
$$
\log\ N(y;\mu,\sigma)=\log\left[{\frac {1}{\sigma {\sqrt {2\pi }}}}\right] + \log \left[e^{-{\frac {1}{2}}\left({\frac {y - \mu}{\sigma }}\right)^{2}}\right]
$$
For the second term you can just drop the logarithm, because $\log\ e^z = z$:
$$
\log \left[e^{-{\frac {1}{2}}\left({\frac {y-\mu }{\sigma }}\right)^{2}}\right] = -{\frac {1}{2}}\left({\frac {y-\mu }{\sigma }}\right)^{2}
$$
In most cases the first term is an additive constant so it could be entirely ignored in optimization leaving us with just the standard MSE error (the second term), with $\mu$ being the regressor's prediction. The derivative of the first term would be 0 anyway.
Here, we want to optimize for $\sigma$, so we can no longer consider this term to be constant! We can, however, get rid of the other constants:
$$
\log\left[{\frac {1}{\sigma {\sqrt {2\pi }}}}\right] = \log\ {\frac {1}{\sigma}} + \log\ {\frac {1}{{\sqrt {2\pi }}}} = \log\ {\frac {1}{\sigma}} + C = -\log\ \sigma + C
$$
Once we negate the simplified log-likelihood and drop the constant, we are quite close to the proposed formula:
$$
\mathcal L = \log\ \sigma + \frac {1}{2}\left({\frac {y-\mu }{\sigma }}\right)^{2}
$$
We can further rearrange to pull the $\frac{1}{2}$ constant out by using $\log\ a^b = b\ \log\ a$ and $a = \sqrt{a^2}$:
$$
\mathcal L = \frac{1}{2} \left[\log\ \sigma^2 + {\frac {(y-\mu)^{2}}{\sigma^{2}}}\right]
$$
Again, this formula applies to a single sample $y$. To apply it to a full dataset or batch one needs to sum over all examples and pull the $\frac{1}{2}$ constant out of the sum for convenience.

One of the interpretations is that your neural network predicts the mean and standard deviation of a normal distribution that your targets are supposed to be coming from. This means that $\mu$ and $\sigma$ should be functions of some input value $x$: $\mu(x)$, $\sigma(x)$ (the neural network).
$$
\mathcal L_D = \frac{1}{2} \sum_{i=0}^{|D|}\left[\log\ \sigma(x_i)^2 + {\frac {(y_i-\mu(x_i))^{2}}{\sigma(x_i)^{2}}}\right]
$$
In the Pytorch formula $y_i$ is the target, $\mu(x_i)$ is the input and $\sigma(x_i)^{2}$ is the variance. The max(var[i], eps) parts are just to avoid numerical errors when computing a logarithm or dividing by small numbers.
A: The loss is just the negative log gaussian pdf up to some constant factor, with the tweak that variance is constrained to be at least $\epsilon$, presumably for numerical reasons (e.g. you wouldn't want to accidentally allow $\sigma^2 = -1$)
