explanation of RBM definition

Question

I'm learning about RBM and I try to understand the notation used for it. We have the input vector $v=(v_i)$ and the output vector $h=(h_j)$, a weight matrix $W=(W_{ij})$ and finally two bias vectors - $a,b$.

Normally for an ANN we would say that - $$h=\sigma(v^TW+a)$$ where $\sigma$ is the network's activation function.

does this relation not hold for RBM? How does $b $ fit in here? Why is it common to talk about $h$ as a vector that stand for itself and not a deterministic result of the input? What part of this definition makes the network stochastic?

Answer 1

A neural network is a function approximator, where as an RBM is a probabilistic graphical model. Although the equations may seem superficially similar, it's best not to confuse the two ideas.

In a probabilistic model, you have probability distributions on the value of the nodes in your graph. So the equation you wrote above for ANN does make sense in the case of RBMs.

In an RBM, the joint distribution over all variables is: $$P(x,h) \propto \exp (b^Tv + a^Th + h^TWv)$$

In the case of binary RBM, if you condition on $x$, you'll end up with $$P(h|x) = \sigma(a+Wx)$$

Obviously, $b$ wouldn't show up in this equation because $x$ is fixed, and $b$ only influences the joint probability through $x$.

So even though this equation is very similar to the ANN equation, the interpretation is quite different.