I am trying to understand the probabilistic concepts behind classification using neural networks, for the goal of incorporating prior information over the target class distributions.

I am failing to assign the terms for conditional probability (bayes equation):

Considering an artificial neural network (ANN) for classification, the softmax output of such network is the probability of a Class given Data and some learned weights = $P(C|D,w)$

Through regularization and weight initialization we can set priors on the weight distribution $P(w)$, and using prior maps on the classes, we can also use probabilities over the classes $P(C)$.

How are these terms coherent with Bayes theorem? Is the output of the neural network a likelihood over classes, or is it a posterior over classes, in which case what would be the likelihood? How do we include priors for classes and weights?

So far all I can write is that given a prior over classes, the likelihood must look like this $L(D|C)$, although the likelihood of seeing Data D given a class C does not make much sense to me if we are predicting classes. The posterior would then be $L(D|C)P(C)$ ~ $P(C|D)$. And I do not know where the weights fit into all of this.

How does the correct bayesian formulation look like for classification with priors over classes?

  • 2
    $\begingroup$ This might not be what you're looking for, but cs.toronto.edu/~radford/ftp/thesis.pdf is the canonical reference for Bayesian neural nets. $\endgroup$ – aleshing May 3 '18 at 2:06
  • $\begingroup$ @marmle thank you I will certainly study that. Alumni from Hinton, looks interesting. So is this a more obscure subject than I thought? I expected a lot of work being done with this, and being more or less a very common topic. But haven't found mention of it in popular machine learning textbooks. $\endgroup$ – hirschme May 3 '18 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.