2
$\begingroup$

In Restricted Boltzman Machines, when collecting the statistics I sometimes heard of fantasy particles being used.

What are these? How are they useful?

$\endgroup$
3
$\begingroup$

Fantasy particles were first introduced by Tielman, using a technique for training RBMs called persistent contrast divergence. See:

Training restricted Boltzmann machines using approximations to the likelihood gradient- Tieleman.

and

Using Fast Weights to Improve Persistent Contrastive Divergence -Tieleman, et. al.

From, A Practical Guide for training Restricted Boltzmann Machines -- Hinton:

A more radical departure from CD1 is called “persistent contrastive divergence” (Tieleman, 2008). Instead of initializing each alternating Gibbs Markov chain at a datavector, which is the essence of CD learning, we keep track of the states of a number of persistent chains or “fantasy particles”. Each persisitent chain has its hidden and visible states updated one (or a few) times after each weight update. The learning signal is then the difference between the pairwise statistics measured on a mini- batch of data and the pairwise statistics measured on the persistent chains. Typically the number of persistent chains is the same as the size of a mini-batch, but there is no good reason for this. The persistent chains mix surprisingly fast because the weight-updates repel each chain from its current state by raising the energy of that state (Tieleman and Hinton, 2009).

See also https://www.cs.toronto.edu/~hinton/csc2535/notes/lec4new.pdf

$\endgroup$
3
  • $\begingroup$ Sorry, what is a Gibbs Markov chain ? $\endgroup$ – octavian Jun 10 '17 at 19:57
  • $\begingroup$ @octavian: en.wikipedia.org/wiki/Gibbs_sampling $\endgroup$ – Alex R. Jun 10 '17 at 21:06
  • $\begingroup$ Do you know of any simple numerical example of RBMs with Fantasy particles? Your explanation is great but a bit too abstract for me to understand. $\endgroup$ – octavian Jun 12 '17 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.