I'm trying to study Boltzmann machines, so I don't undestand this recurrent formulation for the training stage of the weights $w$:

$\Delta w_{ij} = E_{data} (v_i h_j ) − E_{model} (v_i h_j )$

all references tell that $E_{data}$ is the expectation observed in the training set while $E_{model}$ is "that same expectation under the distribution defined by the model"; I don't understand what is this "expectation of the model" and why is intractable; is there a clear reference to understand this concept that is still unclared to me?


The expectation of the model, which refers to that deriving from the negative phase of learning where you gibbs sample freely across all neurons, is intractable because the partition function is intractable. It is intractable because you need expectation over hidden AND visible units (the model) because you have to make an exponential sum over both.

Because it is intractable you have estimate the maximum likelihood gradient with monte carlo methods. So you just take the values once the markov chain is burnt in as an estimate of that intractable computation.

  • $\begingroup$ It is unclear what you are talking about and how it relates to the question. $\endgroup$ – Michael R. Chernick Apr 23 '17 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.