# What is meant by the term “convergence” in Restricted Boltzmann Machine?

I have come across the term "convergence" in training RBM. Can someone give a brief definition / explanation of it?

Depending on which neural net (NN) you are using and what you are doing, NN tries to minimize some cost function $f(W)$ where $W$ are weights. For example, for autoencoders the cost function involves reconstruction error. If you are doing logistic regression, the cost function would be related to classification error. NN tries to find a $W$ that minimizes the cost.

Generally in optimization two quantities are monitored for convergence: $f(W)$ and $W$. If you observe either your cost $f$ or your parameter $W$ does not vary much from one iteration ($iter-1$) to the next ($iter$), the convergence has happened and the loop stops:

$|f(W_{iter}) - f(W_{iter-1})| < \epsilon$
OR $||W_{iter} - W_{iter-1}|| < \epsilon$

That mean that NN in finite iterations (epochs) can approximate local/global minimum of function as close as you need. In math term you can think as:

$\mid W_{real\ minimum} - W_{approximated\ minimum}\mid \leq \epsilon$

Where $\epsilon$ is some small value which allows you to adjust the similarity of two values.

• The thing is that we do not know the $W_{real \ \ minimum}$ so above cannot be used as the convergence criterion. Maybe you meant $||W_{iter-1} - W_{iter}|| \leq \epsilon$ meaning that the parameter to be approximated has converged. – Seeda Feb 5 '15 at 18:47
• Yes, real minimum is unknown and it's just a comparison method for convergence. – itdxer Feb 6 '15 at 8:32
• The first question was about NN, where we have also non-iterative algorithm. But now question is different:) – itdxer Feb 6 '15 at 9:39