2
$\begingroup$

Is it possible to chose the parameters of a RBM to maximize the likelihood of the observed data?

(I follow the notation of the deeplearning tutorial ). Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$. The probability of $x$ is: \begin{equation} P(x) = \sum_h P(x,h) = \sum_h \frac{e^{-E(x,h)}}{Z}. \end{equation} In order to maximize the probability of $x$ conditional on the parameters of the model $\theta$ one could maximize the sum of the derivatives of the log-likelihood: \begin{equation} \sum_{x_i} \frac{\partial \log p(x_i)}{\partial \theta} \end{equation} There exists many different ways to approximate the derivative of the log-likelihood to facilitate (permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $\theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as: \begin{equation} \theta_{i+1} = \theta_{i} - \eta_i \frac{\partial \log p(x_i)}{\partial \theta_i} \end{equation} The practice I learned in Hinton et al. (2006) and in Tieleman (2008) (I'm sorry: I cannot link to that paper due to insufficient reputation) is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.