Questions tagged [restricted-boltzmann-machine]
a Restricted Boltzmann Machine (RBM) is a kind of artificial neural network.
140 questions
18
votes
2
answers
6k
views
What are the works of Hopfield and Hinton that enable machine learning with neural networks, as noted in the physics Nobel award statement?
On October 9, 2024, the Nobel Foundation announced the Nobel Prize in Physics 2024 with the following statement of merit:
for foundational discoveries and inventions that enable machine learning with ...
0
votes
0
answers
20
views
Sampling Gauss-Bernoulli RBM
In the 2018 paper Stein Variational Gradient Descent Without Gradient the authors analyze the sampling performance of their algorithm on multiple benchmarks. One of them is sampling from a Gauss-...
- 11
2
votes
0
answers
32
views
Notation for calculating $p(v)$ marginalising over $h$ in an Restricted Boltzmann Machine
I am working through equation 22 in Introduction to Boltzmann Machines
I am a little confused with the notation, in particular in the line:
As I understand it, we want the probability of a specific ...
- 143
2
votes
0
answers
105
views
Restricted Boltzmann Machine: W matrix visualization results after training MNIST images and Pseudo-log-likelihood
I am implementing RBM from scratch using Tensorflow and after training my RBM on the MNIST dataset for 200 epochs using Persistent CD with two steps of contrastive divergence, I learn the weights W ...
- 21
0
votes
1
answer
312
views
Intuition behind Energy function in Restricted Boltzmann Machines
What does the Energy function in Restricted Boltzmann Machines represent intuitively?
I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
- 3
0
votes
0
answers
20
views
Detailed mathematical derivation of the energy function and joint probability of restricted Boltzmann machine
I am wondering how the joint distribution of restricted Boltzmann machine (RBM) is mathematically derived. Consider a RBM with input layer $\bf x$ (with bias $\mathbf b$), hidden layer $\bf h$ (with ...
- 231
1
vote
0
answers
48
views
Using -1 instead of 0 in binary Restricted Boltzmann Machines
I am aware of the fact that binary Restricted Boltzmann Machines - which use $0$ or $1$ as states - are asymmetric, in the sense that two states have to be both $1$ for $w_{ij}$ to enter in the energy ...
- 111
0
votes
1
answer
65
views
Training in Restricted boltzman machine
I am having doubt in training part of RBM's. I am confused between whats the difference between training RBM by block gibbs sampling and training RBM using contrastive divergence?
- 21
1
vote
1
answer
194
views
Deep Belief Network
I am a bit confused about deep belief networks.
Should the RBM output be the input to the feed forward neural network for the fine tuning step or just the weights of the neural network have to be ...
- 29
0
votes
3
answers
357
views
Is this overfitting?
The Use Case: We are given three unique, 'ground truth' binary training patterns (not patterns with 'noise'). A machine is to be trained with these three vectors. The requirement is that once trained, ...
- 1
2
votes
0
answers
85
views
Sampling from deep belief networks
DBNs are generative models, and usually you sample by thermalising the deepest layer (as it's a restricted Boltzmann Machine), and then forward propagating a sample towards the visible layer to get a ...
- 130
3
votes
0
answers
210
views
What is difference between attention and restricted-Boltzmann machine (RBM)?
Attention has attracted lots of interest recently. However, when I looked at the details of calculation, I recognize a lot of similarity between RBM and attention. Self-attention (key=value) is ...
- 41
0
votes
0
answers
76
views
Subsequent data transformation in Deep Belief Networks
I have a Deep Belief Network made of 4 Restricted Boltzmann Machines. The lowest level RBM seems to train correctly. The deeper layers however seem not to learn anything at all.
I plot the receptive ...
- 11
0
votes
1
answer
88
views
Computing Gradients for a [-1, 1]-valued RBM
The gradient derivation for a binary-valued RBM with values $\in\{0,1\}$ is well-documented, for example in Goodfellow, et al and here on Cross Validated. However, in some works (e.g., associative ...
- 101
0
votes
0
answers
111
views
Proving RBM similar to feedforward neural network with single latent layer
In an examination, I came across a question where I was asked to prove that RBMs are similar to a feedforward neural network with a single layer.
I have an intuition that the structure is similar (...
- 101
0
votes
1
answer
257
views
What does "stochastic" mean in RBM networks?
I am learning about restricted Boltzmann machines (RBM). It is still unclear to me why it is a stochastic algorithm. I was wondering if some one can explain it with comparison to deterministic ones (...
- 211
8
votes
1
answer
2k
views
Is initializing the weights of autoencoders still a difficult problem?
I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it.
I have been reading different articles. In one of Hinton's papers (2006)...
- 211
1
vote
1
answer
1k
views
Energy function of Restricted Boltzmann Machine (RBM)
The energy function for RBM (Restricted Boltzmann Machine) is defined as
$$
E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i
$$
with the joint distribution
$$
\tag{1}
p(...
- 127
1
vote
1
answer
278
views
Deep learning book RBM Equition 20.15 how is the conditional distribution derived?
I was reading the DL textbook from here. I understand how the probability of individual elements of hidden layer h gets derived. However, I could not understand the equation (20.15), which derive the ...
- 109
1
vote
1
answer
363
views
Feature selection using Restricted Boltzmann Machine
I am new in the field of RBMs, DBMs and I cannot understand some things.
I came across the idea of feature selection using RBMs (or Deep Belief Networks).
Although the Hidden nodes which make new ...
0
votes
1
answer
177
views
Can I use Free Energy as reconstruction error when I use RBM to anomaly detection?
Can I use Free Energy as reconstruction error when I use RBM to anomaly detection?
If Free Energy of a sample more a threshold, can I regard it an outlier?
How to explain Free Energy of RBM?
0
votes
1
answer
157
views
Why a Deep Belief Network has connection that points to the input layer?
Supposing to have a 3-layer DBN. I don't understand the specific reason for which the connections between the top two layers are undirected and the connections between all other layers are directed. ...
- 15
0
votes
1
answer
23
views
Randomize dataset for Restricted Boltmann Machines
Suppose I want to train a RBM (or even a DBN architecture) and then fine-tune the parameter training a Feedforward NN. In my case the dataset is composed of time series, so in principle there is a ...
- 15
0
votes
1
answer
102
views
RBM to predict multiple output continuous response
0
I came across a paper where they are predicting GPA/Grades of different courses for a candidate using RBM.
My question is can we apply RBM for a problem where the output needs to be continuous and ...
- 1
1
vote
1
answer
454
views
Feature extraction vs Fine tuning with Restricted Boltmann Machines
I am reading a paper which uses a Restricted Boltzmann Machine to extract features from a dataset in an unsupervised way and then use those features to train a classifier (they use SVM but it could be ...
- 15
0
votes
1
answer
83
views
Why can a Restricted Boltzmann Machine reconstruct using same weights?
Commonly NN (autoencoders) use a set of weights in the reduction process and another in the reconstruction process.
But a RBM uses the same weights in construction and reconstruction process.
I don't ...
- 53
1
vote
1
answer
307
views
Stacked RBM vs. Bernoulli RBM
I'm requested to implement mnist project using stacked RBM. I didn't find any implementation of RBM in keras or tensorflow. However, there is a Bernoulli RBM in sklearn.
Could you please guide me if ...
- 499
5
votes
2
answers
2k
views
Why can't we use backpropagation and gradient descent on a Restricted Boltzmann Machine
Can someone please explain why we cannot use the backpropagation algorithm and gradient descent to train a Restricted Boltzmann Machine. In other words, why can't we train an RBM in the same manner ...
- 1,582
4
votes
0
answers
131
views
Can someone explain in layman terms how the Contrastive Divergence algorithm works step by step?
I am interested in learning about Restricted Boltzmann Machines (RBMs), but I have trouble understanding how an RBM is trained using contrastive divergence. There are only few papers on this topic and ...
- 211
0
votes
1
answer
354
views
Summary of Pre-Training a Neural Network with Stacks of RBMs
I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done.
Say you have a basic neural network with a ...
- 133
0
votes
1
answer
46
views
Probability of the input features in Boltzmann Machine
On the youtube lecture at 8:15(Restricted Boltzmann machine - free energy) professor mentioned, that we want to make the probability of the observed features to be big.
However I didn't understand ...
- 135
3
votes
1
answer
240
views
Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence
In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
- 81
0
votes
1
answer
206
views
Isn't computing the "tractable error" in Restricted Boltzmann Machines (RBM) intractable?
Let $v \in \{0,1\}^M$ be the visible layer, $h \in \{0,1\}^N$ be the hidden layer, where $M$ and $N$ are natural numbers. Given the biases $b \in \Re^M$, $c \in \Re^N$ and weights $W \in \Re^{M \times ...
5
votes
1
answer
2k
views
Restricted Boltzmann Machines vs GAN
Can someone please tell me how RBMs and GANs compare to each other? I know that the community is more excited over GANs than RBMs at the moment. I guess it's because GANs produce better results? My ...
- 225
0
votes
1
answer
151
views
explanation of RBM definition
I'm learning about RBM and I try to understand the notation used for it. We have the input vector $v=(v_i)$ and the output vector $h=(h_j)$, a weight matrix $W=(W_{ij})$ and finally two bias vectors - ...
- 161
3
votes
1
answer
659
views
Why do we want (Restricted) Boltzmann Machines to be stochastic?
Boltzmann machines are stochastic recurrent neural networks with a specific architecture, and they are interesting in several contexts.
They are stochastic in the sense that the sampling procedure ...
- 393
2
votes
0
answers
147
views
Do stacked RBM's have any benefits/advantages over CNN?
Do stacked RBM's have any benefits/advantages over CNN?
If the concern is about face recognition.
- 21
7
votes
1
answer
901
views
Why is MCMC not reliable when compared to stochastic gradient descent?
I came across the following quote on enter link description here
if we made MCMC as reliable as stochastic gradient descent now is for
deep networks, that could mean a resurgence of more explicit
...
- 305
8
votes
2
answers
2k
views
Restricted Boltzmann Machines - Understanding contrastive divergence vs. ML learning
I've read many articles about RBMs as well as Hinton's original paper about the CD algorithm (at least roughly), but I still don't get the big picture. I understand CD conceptually and I've already ...
- 205
1
vote
1
answer
2k
views
scikit's RBM pseudo likelihood calculation
According to scikit-learn's documentation given here, the pseudo likelihood is calculated by;
Computing the free energy on X, then on a randomly corrupted version of X, and returns the log of the ...
- 113
1
vote
2
answers
2k
views
What does it mean to sample from an RBM?
In restricted Boltzmann machines we ofter say that given $v$ we sample $h$, and also that given $h$ we sample $v$.
Can someone explain the physical meaning of this, considering a MNIST image as ...
- 17
-1
votes
1
answer
1k
views
KL divergence derivation
I want to understand KL divergene. Can someone please explain why we need inequality lnx
- 17
0
votes
0
answers
208
views
partition function therm in the derivation of log likelihood in RBM?
In the following derivation when we take derivative of second term why the summation is only over h although it should be be over v and h because we are dealing with Z.
- 17
0
votes
1
answer
390
views
Conditional probability expression in restricted boltzmann machine? How to get p(h=1|v)?
I'm reading a book on Deep learning and having trouble in deriving an expression
Can someone please explain how to go from equation(20.10) to (20.11)? How summation term is converting into product ...
- 17
3
votes
1
answer
174
views
How does a hidden unit in a Boltzmann Machine differ from a hidden unit in a Neural Network?
I was reading about Boltzmann Machines, and I found this:
The state of a hidden unit in a Boltzmann Machine is a random variable, but in a Neural Net it is a deterministic function of the inputs.
...
- 1,199
1
vote
1
answer
1k
views
Why is the log likelihood used for the loss function in an RBM
In an RBM, the loss function is defined like this:
Why are we using the log likelihood function? How does that measure the error?
- 1,199
2
votes
1
answer
505
views
What are fantasy particles in RBMs
In Restricted Boltzman Machines, when collecting the statistics I sometimes heard of fantasy particles being used.
What are these? How are they useful?
- 1,199
1
vote
0
answers
3k
views
Understanding a simple example of restricted Boltzmann machine (RBM)
I am trying to get an intuitive idea of RBMs out of curiosity, and using a simple example on youtube based on preferences for different sports, which denote profiles roughly corresponding to ...
- 26.9k
1
vote
1
answer
2k
views
Restricted Boltzmann Machine: how to implement Gaussian visible units
Please, I'm a newby with Restricted Boltzmann Machine, I'm a psychologist (not very good with math) and and I've some confusion about the use of Gaussian visible units.
Note I'm working on the ...
2
votes
0
answers
192
views
Visible Layer Bias in Restricted Boltzmann Machines
In Neural Networks the bias term of the hidden units can be considered a threshold for the node to fire. This is how neurons basically work in the brain as well. In RBMs, also a bias for the visible ...
- 525