5
$\begingroup$

This might be a stupid question but this question is bugging me for a long time. When I first started working with neural networks we usually created a neural network which output vector of number values. We then used crossentropy or MSE loss for training classification and regression models respectively.

But now many generative models assumes that the output of neural network as a probability distribution and uses loss function such as Kullback Liebler (KL) Divergence, Jensen-Shannon divergence to train a model, example: GANs, VAEs etc.

Why in some training cases the outputs are assumed to be a probability distribution? What is the advantages and disadvantages of this assumption?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Your question appears to mix some concepts. A classification probability is a probability by construction (it satisfies the Kolmogorov axioms: non-negative, measure 1, $\sigma$-additive). But the appearance of KL divergence appearing in a VAE is about constraining the estimates of the parameters of a probability distribution. $\endgroup$
    – Sycorax ♦
    Dec 2, 2021 at 16:27
  • $\begingroup$ Nice question. You have already answered your own question implicitly. In general, generative models by definition requires probability distribution as an output. $\endgroup$
    – patagonicus
    Dec 22, 2021 at 5:44

1 Answer 1

Reset to default
3
$\begingroup$

When using cross-entropy for classification tasks, that can also be viewed as a conditional probability distribution (categorical - the output values are the parameters giving the probability of each catageory) and when using MSE for regression that is also specifying a conditional probability distribution (the conditional mean of a Gaussian distribution - but with a fixed variance). The loss function is a good way of incorporating expert knowledge in the design of the model. I have found this useful, for example in modelling the conditional variance of the response variable, which is not constant in some applications.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.