Questions tagged [restricted-boltzmann-machine]

a Restricted Boltzmann Machine (RBM) is a kind of artificial neural network.

Filter by
Sorted by
Tagged with
2 votes
0 answers
47 views

Restricted Boltzmann Machine: W matrix visualization results after training MNIST images and Pseudo-log-likelihood

I am implementing RBM from scratch using Tensorflow and after training my RBM on the MNIST dataset for 200 epochs using Persistent CD with two steps of contrastive divergence, I learn the weights W ...
user avatar
  • 21
0 votes
1 answer
53 views

Intuition behind Energy function in Restricted Boltzmann Machines

What does the Energy function in Restricted Boltzmann Machines represent intuitively? I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
user avatar
  • 3
0 votes
0 answers
19 views

How to calculate the conditional probabilities for a third-order restricted Boltzmann machine?

For a two-order restricted Boltzmann machine with visible layer $\mathbf x$ and hidden layer $\mathbf h$, the conditional distribution can be written as: $$p(\mathbf h|\mathbf x)=sigmoid(\mathbf{Wx}+\...
user avatar
  • 181
0 votes
0 answers
12 views

Detailed mathematical derivation of the energy function and joint probability of restricted Boltzmann machine

I am wondering how the joint distribution of restricted Boltzmann machine (RBM) is mathematically derived. Consider a RBM with input layer $\bf x$ (with bias $\mathbf b$), hidden layer $\bf h$ (with ...
user avatar
  • 181
1 vote
0 answers
15 views

Using -1 instead of 0 in binary Restricted Boltzmann Machines

I am aware of the fact that binary Restricted Boltzmann Machines - which use $0$ or $1$ as states - are asymmetric, in the sense that two states have to be both $1$ for $w_{ij}$ to enter in the energy ...
user avatar
  • 111
0 votes
0 answers
22 views

Pseudo log-likelihood computation in Restricted Boltzmann machines [duplicate]

I am writing a RBM and I have a doubt. Searching on the Internet, I found that the pseudo log-likelihood is often computed during training. The code I found is something like the following one. ...
user avatar
  • 111
0 votes
1 answer
47 views

Training in Restricted boltzman machine

I am having doubt in training part of RBM's. I am confused between whats the difference between training RBM by block gibbs sampling and training RBM using contrastive divergence?
user avatar
  • 21
1 vote
1 answer
103 views

Deep Belief Network

I am a bit confused about deep belief networks. Should the RBM output be the input to the feed forward neural network for the fine tuning step or just the weights of the neural network have to be ...
user avatar
  • 29
0 votes
3 answers
183 views

Is this overfitting?

The Use Case: We are given three unique, 'ground truth' binary training patterns (not patterns with 'noise'). A machine is to be trained with these three vectors. The requirement is that once trained, ...
user avatar
2 votes
0 answers
56 views

Sampling from deep belief networks

DBNs are generative models, and usually you sample by thermalising the deepest layer (as it's a restricted Boltzmann Machine), and then forward propagating a sample towards the visible layer to get a ...
user avatar
  • 118
1 vote
0 answers
73 views

What is difference between attention and restricted-Boltzmann machine (RBM)?

Attention has attracted lots of interest recently. However, when I looked at the details of calculation, I recognize a lot of similarity between RBM and attention. Self-attention (key=value) is ...
user avatar
0 votes
0 answers
59 views

Subsequent data transformation in Deep Belief Networks

I have a Deep Belief Network made of 4 Restricted Boltzmann Machines. The lowest level RBM seems to train correctly. The deeper layers however seem not to learn anything at all. I plot the receptive ...
user avatar
  • 11
0 votes
1 answer
38 views

Computing Gradients for a [-1, 1]-valued RBM

The gradient derivation for a binary-valued RBM with values $\in\{0,1\}$ is well-documented, for example in Goodfellow, et al and here on Cross Validated. However, in some works (e.g., associative ...
user avatar
0 votes
0 answers
40 views

Proving RBM similar to feedforward neural network with single latent layer

In an examination, I came across a question where I was asked to prove that RBMs are similar to a feedforward neural network with a single layer. I have an intuition that the structure is similar (...
user avatar
  • 101
0 votes
1 answer
159 views

What does "stochastic" mean in RBM networks?

I am learning about restricted Boltzmann machines (RBM). It is still unclear to me why it is a stochastic algorithm. I was wondering if some one can explain it with comparison to deterministic ones (...
user avatar
  • 211
8 votes
1 answer
834 views

Is initializing the weights of autoencoders still a difficult problem?

I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it. I have been reading different articles. In one of Hinton's papers (2006)...
user avatar
  • 211
1 vote
1 answer
718 views

Energy function of Restricted Boltzmann Machine (RBM)

The energy function for RBM (Restricted Boltzmann Machine) is defined as $$ E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i $$ with the joint distribution $$ \tag{1} p(...
user avatar
  • 127
1 vote
1 answer
110 views

Deep learning book RBM Equition 20.15 how is the conditional distribution derived?

I was reading the DL textbook from here. I understand how the probability of individual elements of hidden layer h gets derived. However, I could not understand the equation (20.15), which derive the ...
user avatar
  • 79
0 votes
1 answer
255 views

Feature selection using Restricted Boltzmann Machine

I am new in the field of RBMs, DBMs and I cannot understand some things. I came across the idea of feature selection using RBMs (or Deep Belief Networks). Although the Hidden nodes which make new ...
user avatar
0 votes
1 answer
97 views

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection?

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection? If Free Energy of a sample more a threshold, can I regard it an outlier? How to explain Free Energy of RBM?
user avatar
0 votes
1 answer
94 views

Why a Deep Belief Network has connection that points to the input layer?

Supposing to have a 3-layer DBN. I don't understand the specific reason for which the connections between the top two layers are undirected and the connections between all other layers are directed. ...
user avatar
0 votes
1 answer
21 views

Randomize dataset for Restricted Boltmann Machines

Suppose I want to train a RBM (or even a DBN architecture) and then fine-tune the parameter training a Feedforward NN. In my case the dataset is composed of time series, so in principle there is a ...
user avatar
0 votes
1 answer
54 views

RBM to predict multiple output continuous response

0 I came across a paper where they are predicting GPA/Grades of different courses for a candidate using RBM. My question is can we apply RBM for a problem where the output needs to be continuous and ...
user avatar
  • 1
1 vote
1 answer
393 views

Feature extraction vs Fine tuning with Restricted Boltmann Machines

I am reading a paper which uses a Restricted Boltzmann Machine to extract features from a dataset in an unsupervised way and then use those features to train a classifier (they use SVM but it could be ...
user avatar
0 votes
1 answer
53 views

Why can a Restricted Boltzmann Machine reconstruct using same weights?

Commonly NN (autoencoders) use a set of weights in the reduction process and another in the reconstruction process. But a RBM uses the same weights in construction and reconstruction process. I don't ...
user avatar
  • 53
1 vote
1 answer
239 views

Stacked RBM vs. Bernoulli RBM

I'm requested to implement mnist project using stacked RBM. I didn't find any implementation of RBM in keras or tensorflow. However, there is a Bernoulli RBM in sklearn. Could you please guide me if ...
user avatar
  • 479
5 votes
2 answers
976 views

Why can't we use backpropagation and gradient descent on a Restricted Boltzmann Machine

Can someone please explain why we cannot use the backpropagation algorithm and gradient descent to train a Restricted Boltzmann Machine. In other words, why can't we train an RBM in the same manner ...
user avatar
  • 1,118
3 votes
0 answers
117 views

Can someone explain in layman terms how the Contrastive Divergence algorithm works step by step?

I am interested in learning about Restricted Boltzmann Machines (RBMs), but I have trouble understanding how an RBM is trained using contrastive divergence. There are only few papers on this topic and ...
user avatar
0 votes
1 answer
266 views

Summary of Pre-Training a Neural Network with Stacks of RBMs

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done. Say you have a basic neural network with a ...
user avatar
  • 133
0 votes
1 answer
39 views

Probability of the input features in Boltzmann Machine

On the youtube lecture at 8:15(Restricted Boltzmann machine - free energy) professor mentioned, that we want to make the probability of the observed features to be big. However I didn't understand ...
user avatar
  • 135
3 votes
1 answer
184 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
user avatar
  • 61
0 votes
1 answer
130 views

Isn't computing the "tractable error" in Restricted Boltzmann Machines (RBM) intractable?

Let $v \in \{0,1\}^M$ be the visible layer, $h \in \{0,1\}^N$ be the hidden layer, where $M$ and $N$ are natural numbers. Given the biases $b \in \Re^M$, $c \in \Re^N$ and weights $W \in \Re^{M \times ...
user avatar
5 votes
1 answer
2k views

Restricted Boltzmann Machines vs GAN

Can someone please tell me how RBMs and GANs compare to each other? I know that the community is more excited over GANs than RBMs at the moment. I guess it's because GANs produce better results? My ...
user avatar
  • 225
0 votes
1 answer
136 views

explanation of RBM definition

I'm learning about RBM and I try to understand the notation used for it. We have the input vector $v=(v_i)$ and the output vector $h=(h_j)$, a weight matrix $W=(W_{ij})$ and finally two bias vectors - ...
user avatar
  • 161
3 votes
1 answer
596 views

Why do we want (Restricted) Boltzmann Machines to be stochastic?

Boltzmann machines are stochastic recurrent neural networks with a specific architecture, and they are interesting in several contexts. They are stochastic in the sense that the sampling procedure ...
user avatar
  • 225
2 votes
0 answers
124 views

Do stacked RBM's have any benefits/advantages over CNN?

Do stacked RBM's have any benefits/advantages over CNN? If the concern is about face recognition.
user avatar
7 votes
1 answer
538 views

Why is MCMC not reliable when compared to stochastic gradient descent?

I came across the following quote on enter link description here if we made MCMC as reliable as stochastic gradient descent now is for deep networks, that could mean a resurgence of more explicit ...
user avatar
  • 265
6 votes
2 answers
864 views

Restricted Boltzmann Machines - Understanding contrastive divergence vs. ML learning

I've read many articles about RBMs as well as Hinton's original paper about the CD algorithm (at least roughly), but I still don't get the big picture. I understand CD conceptually and I've already ...
user avatar
1 vote
1 answer
1k views

scikit's RBM pseudo likelihood calculation

According to scikit-learn's documentation given here, the pseudo likelihood is calculated by; Computing the free energy on X, then on a randomly corrupted version of X, and returns the log of the ...
user avatar
1 vote
2 answers
1k views

What does it mean to sample from an RBM?

In restricted Boltzmann machines we ofter say that given $v$ we sample $h$, and also that given $h$ we sample $v$. Can someone explain the physical meaning of this, considering a MNIST image as ...
user avatar
  • 17
-1 votes
1 answer
976 views

KL divergence derivation

I want to understand KL divergene. Can someone please explain why we need inequality lnx
user avatar
  • 17
0 votes
0 answers
159 views

partition function therm in the derivation of log likelihood in RBM?

In the following derivation when we take derivative of second term why the summation is only over h although it should be be over v and h because we are dealing with Z.
user avatar
  • 17
0 votes
1 answer
275 views

Conditional probability expression in restricted boltzmann machine? How to get p(h=1|v)?

I'm reading a book on Deep learning and having trouble in deriving an expression Can someone please explain how to go from equation(20.10) to (20.11)? How summation term is converting into product ...
user avatar
  • 17
3 votes
1 answer
157 views

How does a hidden unit in a Boltzmann Machine differ from a hidden unit in a Neural Network?

I was reading about Boltzmann Machines, and I found this: The state of a hidden unit in a Boltzmann Machine is a random variable, but in a Neural Net it is a deterministic function of the inputs. ...
user avatar
  • 1,009
0 votes
1 answer
892 views

Why is the log likelihood used for the loss function in an RBM

In an RBM, the loss function is defined like this: Why are we using the log likelihood function? How does that measure the error?
user avatar
  • 1,009
2 votes
1 answer
453 views

What are fantasy particles in RBMs

In Restricted Boltzman Machines, when collecting the statistics I sometimes heard of fantasy particles being used. What are these? How are they useful?
user avatar
  • 1,009
1 vote
0 answers
3k views

Understanding a simple example of restricted Boltzmann machine (RBM)

I am trying to get an intuitive idea of RBMs out of curiosity, and using a simple example on youtube based on preferences for different sports, which denote profiles roughly corresponding to ...
user avatar
1 vote
1 answer
1k views

Restricted Boltzmann Machine: how to implement Gaussian visible units

Please, I'm a newby with Restricted Boltzmann Machine, I'm a psychologist (not very good with math) and and I've some confusion about the use of Gaussian visible units. Note I'm working on the ...
user avatar
2 votes
0 answers
181 views

Visible Layer Bias in Restricted Boltzmann Machines

In Neural Networks the bias term of the hidden units can be considered a threshold for the node to fire. This is how neurons basically work in the brain as well. In RBMs, also a bias for the visible ...
user avatar
  • 455
4 votes
1 answer
277 views

ReLUs improve Restricted Boltzmann Machines

This question is about the use of Rectified Linear Units as hidden units in Restricted Boltzmann Machines. In Nair and Hinton's paper, using ReLUs as hidden units is proposed. In section 1, they ...
user avatar
  • 212