Questions tagged [deep-belief-networks]

A type of deep neural network architecture that allows layer-wise unsupervised pre-training.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

What are the applications of RBM and why do we choose RBM for them?

I was wondering what the applications of RBM are. In addition, why do we choose RBM in each of those applications. For example, in some cases both RBMs and auto-encoders can be used, but we may ...
0
votes
0answers
72 views

Does Neural Network or Deep Learning consider correlation among variables? [duplicate]

If there are any correlations among variables, do I need to think of choosing least correlated variables for neural network or deep learning? I am just wondering because Regression Analysis care about ...
2
votes
0answers
257 views

A paper that proves using the latent features of RBM as input to logistic regression?

I'm looking for a paper that includes a proof that simply training a Restricted Boltzmann Machine and then using the latent features as input to a logistic regression classifier is a correct thing and ...
0
votes
1answer
51 views

Feature selection using Restricted Boltzmann Machine

I am new in the field of RBMs, DBMs and I cannot understand some things. I came across the idea of feature selection using RBMs (or Deep Belief Networks). Although the Hidden nodes which make new ...
0
votes
1answer
23 views

Why a Deep Belief Network has connection that points to the input layer?

Supposing to have a 3-layer DBN. I don't understand the specific reason for which the connections between the top two layers are undirected and the connections between all other layers are directed. ...
9
votes
3answers
8k views

Feature selection using deep learning?

I want to calculate the importance of each input feature using deep model. But I found only one paper about feature selection using deep learning - deep feature selection. They insert a layer of ...
3
votes
2answers
545 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
1
vote
0answers
99 views

How to calculate associated degree of freedom of linked nodes in graph

I am working on text analytics and building a knowledge graph with high frequency entities (noun chunks) as graph nodes and their linkage between co-occurrence in a sentence as edges. I am able to ...
0
votes
1answer
1k views

Deep belief networks: supervised or unsupervised?

I want to know whether a Deep Belief Network (or DBN) is a supervised learning algorithm or an unsupervised learning algorithm? After lot of research into DBN working I am confused at this very ...
3
votes
1answer
104 views

What is an “undirected associative memory” in Hinton et al 2006?

In A fast learning algorithm for deep belief nets, the authors use the term "undirected associative memory". I am not sure what they are referring to, and unfortunately Google searches for this term ...
10
votes
4answers
11k views

difference between neural network and deep learning

In terms of the difference between neural network and deep learning, we can list several items, such as more layers are included, massive data set, powerful computer hardware to make training ...
3
votes
1answer
589 views

Do I have to normalize the input vector for RNN if it only consists of 0 and 1?

Yan LeCun's paper "Efficient Backprop" indicates that the average of each input variable over training set should be close to zero. If the input variables are all categorical variables and encoded ...
3
votes
0answers
41 views

Choosing the number of hidden layers and nodes in a Deep Belief Network

What are the recent advances and current best practices in choosing the number and size of stacked Restricted Boltzmann Machines in Deep Belief Networks ?
7
votes
1answer
2k views

Are Restricted Boltzmann Machines better than Stacked Auto encoders and why?

So I'm learning about deep learning. I first learned about stacked auto-encoders and now I'm learning about Restricted Boltzmann Machines. However non in the papers/tutorials I read I found them ...
22
votes
5answers
5k views
5
votes
2answers
12k views

In convolutional neural networks, how to prevent the overfitting?

Given certain amount of labeled data, we define the net structure, such as number of layers, types of layers, the number of convolutional layers, the number of pooling layers, etc. And train the ...
1
vote
0answers
164 views

Is this a really a belief propagation problem?

BACKGROUND This is basically a reputation problem that involves a set of interacting entities $e_i$. Each entity has, in principle, a reputation vector $\vec{b}_i$. That reputation depends on what ...
3
votes
0answers
69 views

State of the Art Status of Deep Boltzmann Machine and Pretraining

I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models: Are DBM and DBN totally outdated? I ...
3
votes
1answer
1k views
63
votes
3answers
48k views

What is the difference between a neural network and a deep belief network?

I am getting the impression that when people are referring to a 'deep belief' network that this is basically a neural network but very large. Is this correct or does a deep belief network also imply ...
11
votes
2answers
2k views

How to understand a convolutional deep belief network for audio classification?

In "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations" by Lee et. al.(PDF) Convolutional DBN's are proposed. Also the method is evaluated for image ...
2
votes
0answers
103 views

Do stacked RBM's have any benefits/advantages over CNN?

Do stacked RBM's have any benefits/advantages over CNN? If the concern is about face recognition.
0
votes
1answer
27 views

Is there any papers or blogs that discuss the effect of embedding layer dimensionality?

Is there any paper or blog that discuss the effect of the dimensionality of embedding layers? The Embedding layers can be used in deep learning models, like CNN or LSTM.
23
votes
3answers
11k views

What is the architecture of a stacked convolutional autoencoder?

So I am trying to do pretraining on images of humans using convolutional nets. I read the papers (Paper1 and Paper2) and this stackoverflow link, but I am not sure I am understand the structure of the ...
-1
votes
1answer
245 views

2 questions about the functions in the `deepnet` Deep Neural Network package in R

I'm using R to perform the Deep Neural Network. But there are so many packages and functions related to neural networks that I am confused. I am wondering about the following two things. What are ...
2
votes
0answers
239 views

Neural Network converges to a constant [duplicate]

I'm having a similar problem to the following post (Feed-Forward) Neural Networks keep converging to mean. The model is built with Deep Neural Network library in Matlab by Masayuki Tanaka. The ...
0
votes
1answer
55 views

Need advice regarding Deep Learning for predictive model

I have been working with neural networks for generation of a predictive model using a multivariate approach. I have come across Deep Learning (or Deep neural networks) as a tool to enhance the success ...
28
votes
2answers
13k views

Deep belief networks or Deep Boltzmann Machines?

I'm confused. Is there a difference between Deep belief networks and Deep Boltzmann Machines? If so, what's the difference?
2
votes
0answers
125 views

boltzmann machine; from logistic function to boltzmann distribution

I'm trying to understand BM; on this topic, tutorials explain it with two formulas: logistic function for the probabilty of single units $p(unit=1)=\frac{1}{1+e^{-\sum\limits_xwx}}$ and, when the ...
1
vote
0answers
207 views

Understanding Deep Belief Networks!

I have implemented Stacked Autoencoder in tensorflow and I was thinking of implementing Deep Belief Networks using Stacked RBM's. I had started reading about DBN's from various websites and through ...
1
vote
1answer
366 views

Deep Learning for sequences

I want to use deep learning techniques to perform better inference tasks than Hidden Markov Models (which is a shallow model)? I was wondering what is the state-of-the art deep learning model to ...
2
votes
1answer
2k views

The learning rule for sigmoid belief nets

On page 12 of this tutorial, it shows that the learning rule is $\Delta w_{ji} = \epsilon s_j(s_i-p_i)$. Can someone show me how this is derived? I got $(1-p_i)s_j$ instead after computing $\frac{d\...
2
votes
1answer
164 views

Boltzmann machines: learning algorithm

I'm trying to study Boltzmann machines, so I don't undestand this recurrent formulation for the training stage of the weights $w$: $\Delta w_{ij} = E_{data} (v_i h_j ) − E_{model} (v_i h_j )$ all ...
7
votes
1answer
2k views

Why are deep belief networks (DBN) rarely used?

I was reading this book about deep learning by Ian and Aron. In the description of DBN they says DBN has fallen out of favor and is rarely used. Deep belief networks demonstrated that deep ...
21
votes
2answers
2k views

Restricted Boltzmann Machine : how is it used in machine learning?

Background: Yes, Restricted Boltzmann Machine (RBM) CAN be used to initiate the weights of a neural network. Also it CAN be used in a "layer-by-layer" way to build a deep belief network (that is, to ...
54
votes
5answers
40k views

Using deep learning for time series prediction

I'm new in area of deep learning and for me first step was to read interesting articles from deeplearning.net site. In papers about deep learning, Hinton and others mostly talk about applying it to ...
1
vote
0answers
263 views

Clustering of time series using RBMs/DBN?

I have a sequence of actions dataset. There are 10 different actions, but lets say for simplicity that I have a1 and a2 actions. The data are not stationary. For some time we have one distribution of ...
2
votes
0answers
281 views

What are the most important parameters to tune in a deep belief network?

I am trying to create a Deep Belief Network (DBF) for a binary classification problem. The nolearn package provides a good library for implementing them. I see that there are very many parameters to ...
56
votes
8answers
46k views

R libraries for deep learning

I was wondering if there's any good R libraries out there for deep learning neural networks? I know there's the nnet, neuralnet,...
8
votes
2answers
5k views

Sparse Autoencoder [Hyper]parameters

I have just started using the autoencoder package in R. Inputs to the autoencode() function include lambda, beta, rho and epsilon. What are the bounds for these ...
2
votes
1answer
1k views

Deep Learning: What happens after each epoch?

I am trying to understand batch size and epochs, and I found this very helpful. Each epoch is all of the data, lets say 10,000 rows, and the number of batches is the number of groups the epoch is ...
2
votes
1answer
480 views

Why is backpropagation used more for fine-tuning than the up-down algorithm for deep belief networks?

Deep belief networks are pre-trained using RBMs then fine tuned for a supervised learning task. For almost every paper that I have read, I have seen back-propagation used instead of the up-down ...
2
votes
2answers
696 views

How to derive this conditional distribution function for a Restricted Boltzmann Machine?

I am following along Ian Goodfellow's new Deep Learning book and, reading the last chapter, I am confused about equations 20.7-20.9. We have a joint distribution function, $P(v,h)$, and we are ...
0
votes
0answers
193 views

Why is my DBN predict only 2 out of 5 classes? [duplicate]

I'm using the Deeplearning.net DBN tutorial to train my data set. I normalize the feature set to zero-mean-unit-variance. However, I can only get the network to predict 2 out 5 classes even though the ...
2
votes
2answers
1k views

Dimensionality reduction: RBM autoencoders vs. de-noising autoencoders

I am looking at non-linear dimensionality reduction techniques and am currently trying to understand the practical differences between different autoencoder approaches: Can somebody point me to a ...
8
votes
1answer
4k views

Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?

Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ? Or is it so that all record breaking approaches use some form of pre-training ? Is back-...
7
votes
3answers
3k views

How to derive the gradient formula for the Maximum Likelihood in RBM?

I am learning RBM (restricted Boltzmann machine) for deep learning. The log-likelihood of RBM is given as : $$\ln(L(\theta|v))=\ln(p(v|\theta))=\ln\frac{1}{Z}\sum_h e^{-E(v,h)}=\ln\sum_h e^{E(v,h)}-...
2
votes
0answers
1k views

Threshold on tanh or sigmoid in Convolutional neural network

I have read several papers on Convolutional Neural Nets but I am yet to come across any that has used thresholds on tanh or sigmoid to decide whether the neuron will fire or not. Obviously this works ...
2
votes
0answers
187 views

Deep Belief Net applied to Netflix Prize?

In Restricted Boltzmann Machines for Collaborative Filtering Restricted Boltzmann Machines (RBMs) are applied to the Netflix Prize data set. An obvious next step might be to use stacks of RBMs (i.e. ...
2
votes
1answer
2k views

Scale invariance for images

Given that images can be of vastly different resolutions, but neural networks are usually presented as having a fixed number of inputs, what are the standard techniques used to handle the difference ...