4
$\begingroup$

I am trying to read and understand this paper. They are introducing a method inspired by Assumed Density Filtering (ADF) to train a Bayesian neural network. In section 3, where they are explaining their backpropagation model and comparing it to the normal backpropagation, they say, "PBP also has two phases equivalent to the ones of BP. In the first phase, the input data is propagated forward through the network. However, since the weights are now random, the activations produced in each layer are also random and result in (intractable) distributions.".

What has been confusing me is that why the multiplication of a constant by an initial distribution of a weight results in an intractable distribution? what do they mean by the intractability of these distributions? I appreciate any help. I am new to the field of BNNs and have been having a difficult time adjusting myself to the methods introduced in related papers.

$\endgroup$
1

1 Answer 1

2
$\begingroup$

If you have multiple fully connected layers, you end up with a Bayesian network model that has a large number of loops; exact inference in this kind of network structure is difficult. Qualitatively it's that everything ends up being correlated with everything else.

Consider a neural network model structure like below enter image description here Now interpret this figure as a Bayesian network by orienting the edges, say from right to left (i.e. arrows point to the left). In this network structure there are a huge number of overlapping loops. If one wants to execute exact inference using a message passing algorithm, these overlapping loops will result in difficult to evaluate junction trees.

Really the problem is less about the depth of the network than about the width -- each node of the junction tree will correspond to an adjacent pair of layer, so the size (number of variables) for the clique potentials will scale like the width of the network layers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.