In classification problems, the usual negative log-likelihood loss function
is always non-negative, since the $y_i$'s are discrete random variables and, as such, $p(y_i|x_i,\theta) \leq 1$ for all $\theta$. Therefore, the existence of a minimizer $\theta^*$ for $L(\theta)$ is guaranteed.
However, if the $y_i$'s are continuous random variables (as is the case for Gaussian Mixtures), $p(y|x,\theta)$ is a pdf, so it may assume values greater than 1 (as long as it integrates to 1 w.r.t. $y$). Therefore, if we make no assumptions on the family of pdf's $p(y|x,\theta)$, it is not obvious to me that $L(\theta)$ is necessarily lower bounded. Is it?
Moreover, let us suppose that $p(y|x,\theta)$ is a Gaussian Mixture whose parameters are provided by a Mixture Density Network, that is:
where $K$ is the number of components in the mixture and $\alpha_i(x)$, $\mu_i(x)$ and $\sigma^2_i(x)$ are the outputs of a neural network that takes $x$ as input. In this case, $\theta$ represents all the parameters of the neural network to be optimized.
(for more details on Mixture Density Networks please check Bishop, 1994, available at https://publications.aston.ac.uk/373/1/NCRG_94_004.pdf)
Is $L(\theta)$ lower bounded for this particular family $p(y|x,\theta)$? How to prove (or at least have a strong intuition on) that?
These questions came to my mind while I was trying to reproduce the toy example in section 5 of the paper I mention above. Actually, during training, my loss is always decreasing, assuming negative values after about 1000 iterations. Since I do not have a lower bound for the loss, I don't have any idea if the final loss is reasonable or not. Moreover, I am not being able to reproduce the results in the paper (for instance, for $x=0.5$, I get a unimodal distribution instead of a trimodal one).
Thank you in advance.