Questions tagged [rbm]

Restricted Boltzman Machine

Filter by
Sorted by
Tagged with
0
votes
3answers
102 views

Is this overfitting?

The Use Case: We are given three unique, 'ground truth' binary training patterns (not patterns with 'noise'). A machine is to be trained with these three vectors. The requirement is that once trained, ...
2
votes
0answers
24 views

Sampling from deep belief networks

DBNs are generative models, and usually you sample by thermalising the deepest layer (as it's a restricted Boltzmann Machine), and then forward propagating a sample towards the visible layer to get a ...
0
votes
0answers
29 views

What is difference between attention and restricted-Boltzmann machine (RBM)?

Attention has attracted lots of interest recently. However, when I looked at the details of calculation, I recognize a lot of similarity between RBM and attention. Self-attention (key=value) is ...
0
votes
0answers
39 views

Subsequent data transformation in Deep Belief Networks

I have a Deep Belief Network made of 4 Restricted Boltzmann Machines. The lowest level RBM seems to train correctly. The deeper layers however seem not to learn anything at all. I plot the receptive ...
0
votes
1answer
21 views

Computing Gradients for a [-1, 1]-valued RBM

The gradient derivation for a binary-valued RBM with values $\in\{0,1\}$ is well-documented, for example in Goodfellow, et al and here on Cross Validated. However, in some works (e.g., associative ...
0
votes
0answers
25 views

Proving RBM similar to feedforward neural network with single latent layer

In an examination, I came across a question where I was asked to prove that RBMs are similar to a feedforward neural network with a single layer. I have an intuition that the structure is similar (...
0
votes
1answer
72 views

What does “stochastic” mean in RBM networks?

I am learning about restricted Boltzmann machines (RBM). It is still unclear to me why it is a stochastic algorithm. I was wondering if some one can explain it with comparison to deterministic ones (...
8
votes
1answer
241 views

Is initializing the weights of autoencoders still a difficult problem?

I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it. I have been reading different articles. In one of Hinton's papers (2006)...
0
votes
0answers
19 views

What are the applications of RBM and why do we choose RBM for them?

I was wondering what the applications of RBM are. In addition, why do we choose RBM in each of those applications. For example, in some cases both RBMs and auto-encoders can be used, but we may ...
1
vote
1answer
224 views

Energy function of Restricted Boltzmann Machine (RBM)

The energy function for RBM (Restricted Boltzmann Machine) is defined as $$ E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i $$ with the joint distribution $$ \tag{1} p(...
1
vote
1answer
62 views

Deep learning book RBM Equition 20.15 how is the conditional distribution derived?

I was reading the DL textbook from here. I understand how the probability of individual elements of hidden layer h gets derived. However, I could not understand the equation (20.15), which derive the ...
0
votes
1answer
128 views

Feature selection using Restricted Boltzmann Machine

I am new in the field of RBMs, DBMs and I cannot understand some things. I came across the idea of feature selection using RBMs (or Deep Belief Networks). Although the Hidden nodes which make new ...
0
votes
1answer
39 views

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection?

Can I use Free Energy as reconstruction error when I use RBM to anomaly detection? If Free Energy of a sample more a threshold, can I regard it an outlier? How to explain Free Energy of RBM?
0
votes
1answer
30 views

Why a Deep Belief Network has connection that points to the input layer?

Supposing to have a 3-layer DBN. I don't understand the specific reason for which the connections between the top two layers are undirected and the connections between all other layers are directed. ...
0
votes
1answer
20 views

Randomize dataset for Restricted Boltmann Machines

Suppose I want to train a RBM (or even a DBN architecture) and then fine-tune the parameter training a Feedforward NN. In my case the dataset is composed of time series, so in principle there is a ...
0
votes
1answer
33 views

RBM to predict multiple output continuous response

0 I came across a paper where they are predicting GPA/Grades of different courses for a candidate using RBM. My question is can we apply RBM for a problem where the output needs to be continuous and ...
1
vote
1answer
269 views

Feature extraction vs Fine tuning with Restricted Boltmann Machines

I am reading a paper which uses a Restricted Boltzmann Machine to extract features from a dataset in an unsupervised way and then use those features to train a classifier (they use SVM but it could be ...
0
votes
1answer
34 views

Why can a Restricted Boltzmann Machine reconstruct using same weights?

Commonly NN (autoencoders) use a set of weights in the reduction process and another in the reconstruction process. But a RBM uses the same weights in construction and reconstruction process. I don't ...
0
votes
1answer
177 views

Stacked RBM vs. Bernoulli RBM

I'm requested to implement mnist project using stacked RBM. I didn't find any implementation of RBM in keras or tensorflow. However, there is a Bernoulli RBM in sklearn. Could you please guide me if ...
4
votes
2answers
571 views

Why can't we use backpropagation and gradient descent on a Restricted Boltzmann Machine

Can someone please explain why we cannot use the backpropagation algorithm and gradient descent to train a Restricted Boltzmann Machine. In other words, why can't we train an RBM in the same manner ...
3
votes
0answers
107 views

Can someone explain in layman terms how the Contrastive Divergence algorithm works step by step?

I am interested in learning about Restricted Boltzmann Machines (RBMs), but I have trouble understanding how an RBM is trained using contrastive divergence. There are only few papers on this topic and ...
0
votes
1answer
198 views

Summary of Pre-Training a Neural Network with Stacks of RBMs

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done. Say you have a basic neural network with a ...
0
votes
1answer
36 views

Probability of the input features in Boltzmann Machine

On the youtube lecture at 8:15(Restricted Boltzmann machine - free energy) professor mentioned, that we want to make the probability of the observed features to be big. However I didn't understand ...
0
votes
0answers
207 views

Training RBM with (normal) contrastive divergence vs persistent contrastive divergence

I implemented RBM by using PyTorch here and trained it with CalTech 101 28x28 Silhouettes dataset containing binary images. I implemented both (naive) contrastive divergence and persistent ...
3
votes
1answer
124 views

Gibbs Sampling vs. Using Raw Probability in Contrastive Divergence

In Hinton's Practical Guide to Training Restricted Boltzmann Machines, Section 3, he discusses different situations in which one should take a sample from the Gibbs sampling process, and other ...
0
votes
1answer
79 views

Isn't computing the “tractable error” in Restricted Boltzmann Machines (RBM) intractable?

Let $v \in \{0,1\}^M$ be the visible layer, $h \in \{0,1\}^N$ be the hidden layer, where $M$ and $N$ are natural numbers. Given the biases $b \in \Re^M$, $c \in \Re^N$ and weights $W \in \Re^{M \times ...
5
votes
1answer
1k views

Restricted Boltzmann Machines vs GAN

Can someone please tell me how RBMs and GANs compare to each other? I know that the community is more excited over GANs than RBMs at the moment. I guess it's because GANs produce better results? My ...
0
votes
1answer
109 views

explanation of RBM definition

I'm learning about RBM and I try to understand the notation used for it. We have the input vector $v=(v_i)$ and the output vector $h=(h_j)$, a weight matrix $W=(W_{ij})$ and finally two bias vectors - ...
2
votes
1answer
490 views

Why do we want (Restricted) Boltzmann Machines to be stochastic?

Boltzmann machines are stochastic recurrent neural networks with a specific architecture, and they are interesting in several contexts. They are stochastic in the sense that the sampling procedure ...
2
votes
0answers
110 views

Do stacked RBM's have any benefits/advantages over CNN?

Do stacked RBM's have any benefits/advantages over CNN? If the concern is about face recognition.
6
votes
1answer
302 views

Why is MCMC not reliable when compared to stochastic gradient descent?

I came across the following quote on enter link description here if we made MCMC as reliable as stochastic gradient descent now is for deep networks, that could mean a resurgence of more explicit ...
5
votes
2answers
594 views

Restricted Boltzmann Machines - Understanding contrastive divergence vs. ML learning

I've read many articles about RBMs as well as Hinton's original paper about the CD algorithm (at least roughly), but I still don't get the big picture. I understand CD conceptually and I've already ...
0
votes
1answer
706 views

scikit's RBM pseudo likelihood calculation

According to scikit-learn's documentation given here, the pseudo likelihood is calculated by; Computing the free energy on X, then on a randomly corrupted version of X, and returns the log of the ...
1
vote
2answers
1k views

What does it mean to sample from an RBM?

In restricted Boltzmann machines we ofter say that given $v$ we sample $h$, and also that given $h$ we sample $v$. Can someone explain the physical meaning of this, considering a MNIST image as ...
-1
votes
1answer
815 views

KL divergence derivation

I want to understand KL divergene. Can someone please explain why we need inequality lnx
0
votes
0answers
127 views

partition function therm in the derivation of log likelihood in RBM?

In the following derivation when we take derivative of second term why the summation is only over h although it should be be over v and h because we are dealing with Z.
0
votes
1answer
214 views

Conditional probability expression in restricted boltzmann machine? How to get p(h=1|v)?

I'm reading a book on Deep learning and having trouble in deriving an expression Can someone please explain how to go from equation(20.10) to (20.11)? How summation term is converting into product ...
2
votes
1answer
147 views

How does a hidden unit in a Boltzmann Machine differ from a hidden unit in a Neural Network?

I was reading about Boltzmann Machines, and I found this: The state of a hidden unit in a Boltzmann Machine is a random variable, but in a Neural Net it is a deterministic function of the inputs. ...
0
votes
1answer
589 views

Why is the log likelihood used for the loss function in an RBM

In an RBM, the loss function is defined like this: Why are we using the log likelihood function? How does that measure the error?
2
votes
1answer
409 views

What are fantasy particles in RBMs

In Restricted Boltzman Machines, when collecting the statistics I sometimes heard of fantasy particles being used. What are these? How are they useful?
1
vote
0answers
3k views

Understanding a simple example of restricted Boltzmann machine (RBM)

I am trying to get an intuitive idea of RBMs out of curiosity, and using a simple example on youtube based on preferences for different sports, which denote profiles roughly corresponding to ...
1
vote
1answer
1k views

Restricted Boltzmann Machine: how to implement Gaussian visible units

Please, I'm a newby with Restricted Boltzmann Machine, I'm a psychologist (not very good with math) and and I've some confusion about the use of Gaussian visible units. Note I'm working on the ...
2
votes
0answers
149 views

Visible Layer Bias in Restricted Boltzmann Machines

In Neural Networks the bias term of the hidden units can be considered a threshold for the node to fire. This is how neurons basically work in the brain as well. In RBMs, also a bias for the visible ...
3
votes
1answer
175 views

ReLUs improve Restricted Boltzmann Machines

This question is about the use of Rectified Linear Units as hidden units in Restricted Boltzmann Machines. In Nair and Hinton's paper, using ReLUs as hidden units is proposed. In section 1, they ...
1
vote
0answers
270 views

Clustering of time series using RBMs/DBN?

I have a sequence of actions dataset. There are 10 different actions, but lets say for simplicity that I have a1 and a2 actions. The data are not stationary. For some time we have one distribution of ...
15
votes
3answers
513 views

What does the “machine” in “support vector machine” and “restricted Boltzmann machine” mean?

Why are they called "machines"? Is there an origin to the word "machine" used in this context? (Like the name "linear programming" can be confusing but we know why it is called "programming.")
2
votes
1answer
2k views

Derivation of Restricted Boltzmann Machine Conditional Probability

I was reading Goodfellow, Bengio, and Courville's Deep Learning and in Chapter 20 there is a derivation of the conditional probability $P(h_j=1|\boldsymbol{v})$, the conditional probability of the $j^{...
3
votes
3answers
3k views

Basic confusion about Restricted Boltzmann Machines (RBM)

As I understand it, the standard restricted Boltzmann machine (RBM) exhibits binary stochastic visible and hidden units. The joint probability of the binary and visible units is given by the Boltzmann ...
1
vote
0answers
82 views

question of sampling in a binary-binary RBM

I've learned that the weights of the RBM defines the joint probability distribution of the visible units and hidden units. So I try a very simple matlab code to prove that. I try two methods to ...
2
votes
1answer
178 views

Boltzmann machines: learning algorithm

I'm trying to study Boltzmann machines, so I don't undestand this recurrent formulation for the training stage of the weights $w$: $\Delta w_{ij} = E_{data} (v_i h_j ) − E_{model} (v_i h_j )$ all ...