0
$\begingroup$

I am wondering how the joint distribution of restricted Boltzmann machine (RBM) is mathematically derived. Consider a RBM with input layer $\bf x$ (with bias $\mathbf b$), hidden layer $\bf h$ (with bias $\bf c$) and connection matrix $\bf W$, most of the tutorials that I've seen first directly define the energy function:

$$E(\mathbf{x},\mathbf{h})=-\mathbf{h}^{T}\mathbf{Wx}-\mathbf{b}^{T}\mathbf{x}-\mathbf{c}^{T}\mathbf{h}$$

and then directly tells me that the joint probability is:

$$p(\mathbf{x},\mathbf{h}) = \frac{e^{-E(\mathbf{x},\mathbf{h})}}{\sum_{x,h} e^{-E(\mathbf{x},\mathbf{h})}}$$

and this makes me hard to understand why they are defined in this way. I know it somehow relates to some conclusions in statistical mechanics, but I have never taken a course in this field.

I hope to know if there is a clear and detailed mathematical proof of the above formulas. Specifically, from just the following conditional probabilities:

$$p(\mathbf{h}|\mathbf{x})=\mathrm{sigmoid}(\mathbf{Wx}+\mathbf{c})$$ $$p(\mathbf{x}|\mathbf{h})=\mathrm{sigmoid}(\mathbf{W}^{T}\mathbf{h}+\mathbf{b})$$

is there a way to derive both the energy function and joint probability shown above?

$\endgroup$
3
  • $\begingroup$ Lookup Boltzmann distribution, that will answer 75% of your question $\endgroup$
    – Aksakal
    Oct 25, 2021 at 12:35
  • $\begingroup$ @Aksakal Already checked the Wikipedia page of Boltzmann distribution, but I am still confused about how the formulas of joint probability and the two conditional probabilities are related to each other. Any other recommendations? $\endgroup$
    – Cloudy
    Oct 25, 2021 at 12:46
  • $\begingroup$ if you understand how logistic regression works in this regard then this paper will help to understand multinomial case: jstor.org/stable/2336391 $\endgroup$
    – Aksakal
    Oct 25, 2021 at 12:52

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.