In classification problems, the usual negative log-likelihood loss function

$L(\theta)=\sum_{i=1}^N -\log(p(y_i|x_i,\theta))$

is always non-negative, since the $y_i$'s are discrete random variables and, as such, $p(y_i|x_i,\theta) \leq 1$ for all $\theta$. Therefore, the existence of a minimizer $\theta^*$ for $L(\theta)$ is guaranteed.

However, if the $y_i$'s are continuous random variables (as is the case for Gaussian Mixtures), $p(y|x,\theta)$ is a pdf, so it may assume values greater than 1 (as long as it integrates to 1 w.r.t. $y$). Therefore, if we make no assumptions on the family of pdf's $p(y|x,\theta)$, it is not obvious to me that $L(\theta)$ is necessarily lower bounded. Is it?

Moreover, let us suppose that $p(y|x,\theta)$ is a Gaussian Mixture whose parameters are provided by a Mixture Density Network, that is:

$p(y|x,\theta)=\sum_{i=1}^K \alpha_i(x)\mathcal{N}(y|\mu_i(x),\sigma^2_i(x))$,

where $K$ is the number of components in the mixture and $\alpha_i(x)$, $\mu_i(x)$ and $\sigma^2_i(x)$ are the outputs of a neural network that takes $x$ as input. In this case, $\theta$ represents all the parameters of the neural network to be optimized.

(for more details on Mixture Density Networks please check Bishop, 1994, available at https://publications.aston.ac.uk/373/1/NCRG_94_004.pdf)

Is $L(\theta)$ lower bounded for this particular family $p(y|x,\theta)$? How to prove (or at least have a strong intuition on) that?

These questions came to my mind while I was trying to reproduce the toy example in section 5 of the paper I mention above. Actually, during training, my loss is always decreasing, assuming negative values after about 1000 iterations. Since I do not have a lower bound for the loss, I don't have any idea if the final loss is reasonable or not. Moreover, I am not being able to reproduce the results in the paper (for instance, for $x=0.5$, I get a unimodal distribution instead of a trimodal one).

Thank you in advance.

  • $\begingroup$ why do you think boundedness will help you determine if you minimum is global instead of local? $\endgroup$ – Taylor Jul 26 '17 at 16:10
  • $\begingroup$ I did not say that. I just argue that boundedness is useful to help you understanding how close you are to a good solution. Moreover, if the loss function goes to $- \infty$, gradients will never converge to zero and will probably explode at some point. When should I stop training? $\endgroup$ – D... Jul 26 '17 at 17:01

Actually, the loss is not lower bounded and the problem is actually ill-posed, since one of the mixture components may collapse in a data point, making the loss decrease to arbitrarily small values (i.e. going to $-\infty$).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.