6
$\begingroup$

I spent some time yesterday learning about Boltzmann machines (and their cousins, restricted Boltzmann machines). They seem interesting and very powerful, but one little thing is bugging me.

The energy function for a Boltzmann machine is (assuming the use of a bias node) $$ E = -\sum_{ij} w_{ij} s_i s_j, $$ where the $w_{ij}$ are the weights, and the $s_i$ are the states of the nodes. Every resource I found used $0$ and $1$ as the possible values for $s_i$, but this seems to make the energy function asymmetric, in the sense that two nodes $i$ and $j$ both have to be in the $1$ state for the $w_{ij}$ term to enter into the energy expression, whereas if just one of them is on, it's the same as them both being off.

But it seems that there's nothing in the derivation of the learning algorithm that requires this. As far as I can see, one could use $-1$ and $1$ instead and the learning algorithm would be exactly the same. In this case the contribution to the energy would depend only on whether the two nodes are in the same state or different ones.

My question is whether this makes any difference, or whether the two possibilities turn out to be equivalent somehow. If they are different, is it just that using $0$ and $1$ turns out to be better in practice? I can imagine this being the case if Boltzmann machines are typically used in cases where $0$ and $1$ have semantically different meanings, but I find it surprising that this issue isn't mentioned in any of the introductory articles I found online.

$\endgroup$
3
  • $\begingroup$ Aren't you just shifting the energy functional by trace W-(sum_(i!=j) w_ij)? $\endgroup$
    – Bort
    Jun 8, 2013 at 8:11
  • $\begingroup$ @Bort I don't believe so, no. Consider a Boltzmann machine with two nodes and a weight of -1 between them. With states in $\{0,1\}$ states, the energy levels of the four possible states are 0, 0, 0, and 1, whereas with $\{-1,1\}$ states they are 1, -1, -1 and 1. The two energy functions are not related by a constant offset. $\endgroup$
    – N. Virgo
    Jun 8, 2013 at 15:59
  • $\begingroup$ Also: openreview.net/forum?id=BkbwJj-OZH $\endgroup$
    – a06e
    May 20, 2022 at 9:55

1 Answer 1

3
$\begingroup$

Yes, there is an asymmetry. I haven't thought deeply about it, but I do know of two papers that relate to the issue: 1, 2

Hope these help!

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.