Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

2,567 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
20
votes
1answer
5k views

Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
10
votes
0answers
3k views

Validation loss increases while Training loss decrease

I am training a model and the accuracy increases in both the training and validation sets. I am using a pre-trained model as my dataset is very small. I am not sure why the loss increases in the ...
9
votes
0answers
5k views

When should I use the Normal distribution or the Uniform distribution when using Xavier initialization?

Xavier initialization seems to be used quite widely now to initialize connection weights in neural networks, especially deep ones (see What are good initial weights in a neural network?). The ...
7
votes
0answers
74 views

Overfitting and underfitting in Neural Networks: Is total number of neurons or number of neurons per layer more relevant?

I have seen posts where the discussion was centered around the effect of big and small total number of neurons in a neural network, especially with respect to the potential of the network to overfit ...
7
votes
0answers
88 views

Is there a ML or DL tool that can learn to detect periodically occurring patterns in a one dimensional time series?

I am trying to create a tool that labels refrigerator temperature readings. A reading is taken every 5 minutes, and its label identifies whether of not it was taken while the refrigerator was ...
7
votes
0answers
703 views

How to evaluate performance of (variational) autoencoders?

Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly ...
7
votes
1answer
515 views

Deep Learning vs Structured Learning

I am interested in the differences between using large, deep learning networks vs Probabilistic graphical models (PGMs), like Random Field models, for structured learning (e.g. on images, or labels of ...
6
votes
0answers
71 views

Choosing the number of hidden layers and nodes in a Deep Belief Network

What are the recent advances and current best practices in choosing the number and size of stacked Restricted Boltzmann Machines in Deep Belief Networks ?
6
votes
0answers
7k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
6
votes
0answers
632 views

How is the loss calculated in a Wasserstein GAN?

I'm trying to implement a Wasserstein GAN according to this blog post: https://myurasov.github.io/2017/09/24/wasserstein-gan-keras.html And it has a wasserstein loss of: ...
6
votes
0answers
733 views

Compatible Function Approximation Theorem in Reinforcement Learning

In the Compatible Function Approximation Theorem, the following condition is required to make the policy gradient to be exact $\nabla J(\theta) = \mathbb{E}_{\pi_{\theta}}\left [\nabla_{\theta}log\pi_{...
5
votes
0answers
44 views

Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
5
votes
0answers
79 views

What does it mean to take the expectation with respect to a probability distribution?

I see this expectation in a lot of machine learning literature: $$\mathbb{E}_{p(\mathbf{x};\mathbf{\theta})}[f(\mathbf{x};\mathbf{\phi})] = \int p(\mathbf{x};\mathbf{\theta}) f(\mathbf{x};\mathbf{\phi}...
5
votes
0answers
117 views

Why aren't neural networks used with RBF activation functions (or other non-monotonic ones)?

In most work I've seen, MLPs (multilayer perceptron, the most typical feedforward neural network) and RBF (radial basis function) networks are compared as distinct models, where MLP neuron outputs $\...
5
votes
0answers
243 views

Understanding Object2Vec

AWS released an interesting feature as part of the SageMaker service called Object2Vec that lets you make an embedding for search out of pretty much anything: documents, users, products, ...
5
votes
0answers
403 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
5
votes
1answer
403 views

Clarification: Are Generative Adversarial Networks an alternative to MCMC sampling?

I have been reading the original Goodfellow, et. al. paper on Generative Adversarial Networks and the way that they can obtain estimates of the posterior distribution of a discriminative network or ...
5
votes
0answers
327 views

what is the mistake of convergence proof in Adam

Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
5
votes
0answers
1k views

How to normalize input data for autoencoders - anomaly detection

I'm building an autoencoder to identify anomalies on numerical data. The input features have different scales (i.e. some take values from 0 to 5, while others can be much much higher) and most of them ...
5
votes
1answer
2k views

Are there networks specialised on object detection for a single class of object?

I want to detect the location of a single class of object, which might occur multiple times in an image. Specifically, this relates to research on detecting brake lights for autonomous vehicles. I ...
5
votes
0answers
110 views

Would gradient boosting machines benefit from adaptive learning rates?

In deep learning, a big deal is made about optimizing an adaptive learning rate. There are numerous popular adaptive learning rate algorithms. The hyperparameters for all of the leading gradient ...
5
votes
0answers
865 views

Why word embeddings learned from word2vec are linearly correlated

I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
5
votes
0answers
675 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
5
votes
0answers
574 views

Is google's wavenet architecture computing a bunch of values that it will never use?

I've been trying to understand the wavenet paper. In order to do so, I am using this implementation that I found on github because it gets good results and it is pretty clear. But I have a question ...
5
votes
0answers
793 views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
5
votes
1answer
2k views

Using Rolling Forecast Origin Resampling in R for Neural Network Time Series

I am new to time series prediction and forecasting with neural networks and am having trouble with cross validation. I am fitting a multivariate time series. I have 236 monthly observations. I am ...
5
votes
1answer
83 views

Does $L_1$ regularization help ameliorate the credit assignment problem in (deep) neural nets?

Caveat: this is just a thought that occurred to my while I was driving to work, so maybe it's not well-considered. One challenge to deep neural networks is the credit assignment problem: a node at ...
5
votes
0answers
526 views

TensorFlow Deep MNIST for Experts tutorial: kernels seem to never learn anything

I'm following Google's TensorFlow Deep MNIST for Experts tutorial. Here is my code: http://pastebin.com/ePktssrn The networks seems to get close to 100% accuracy after about training 1000 steps, ...
5
votes
0answers
187 views

How to fix this implementation of Bayesian regularization for ANNs?

I have implemented the Levenberg-Marquard algorithm (from Hagan's "Artifical Neural Network Design" -- 2014) for a two layer network with 20 neurons in the hidden layer. This network can ...
5
votes
1answer
390 views

Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?

I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font: Our key insight is to exploit the ...
5
votes
0answers
1k views

Recurrent neural network for object tracking & position filtering?

Would a recurrent neural network be appropriate for object tracking tasks? Mainly I will have 3D feature vectors $(x, y, t)$ where $x$ and $y$ are the positions of an object in the image and $t$ is ...
5
votes
0answers
355 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
5
votes
1answer
144 views

Is there any procedure to determine the number of layers of convolution and pooling needs in CNN?

When I want to use Caffe to create my own CNN, how to determine the number of convolution and pooling layer I need is suitable to extract correct features basis? Is there any principal or documents? ...
5
votes
0answers
365 views

What enforces features diversity in RBM?

I'm working on an implementation of a Restricted Boltzman Machine (RBM). I made some tests on the MNIST dataset trying to learn a representation of the digit 2. My inputs are binary images. My aim is ...
5
votes
0answers
430 views

Any implementations of fully recurrent neural networks applied to reinforcement learning?

I've seen a single paper on the topic of adapting fully recurrent networks to a reinforcement learning setting, but according to google scholar its had no citations and no code has been released ...
4
votes
1answer
267 views

Does Attention Help with standard auto-encoders

I understand the use of attention mechanisms in the encoder-decoder for sequence-to-sequence problem such as a language translator. I am just trying to figure out whether it is possible to use ...
4
votes
1answer
37 views

Does gradient descent assume updates of one layer/parameter at a time?

I read the following in "Deep Learning", from Goodfellow et al (Chapter 8, page 313): The gradient tells how to update each parameter, under the assumption that the other layers do not change. In ...
4
votes
0answers
29 views

Why is homoscedasticity (homogeneity of variance) important in neural network layers?

I'm studying the famous Xavier initialization paper (Understanding the Difficulty of Training Deep Feedforward Neural Networks (Glorot and Bengio, 2010)) and had a question. When they explain the ...
4
votes
0answers
25 views

What assumptions about time series data are neccessary to use a stateless LSTM?

Basically I am trying to model a time series using an LSTM layer, and I was wondering whether I should be using a stateful or stateless LSTM layer. More specifically I was wondering what assumptions ...
4
votes
0answers
1k views

Advantage of RMSProp over Adam?

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of ...
4
votes
0answers
286 views

Difference between Shapley values and SHAP

The Paper regarding die SHAP value gives a formula for the Shapley Values in (4) and for SHAP values apparently (?) in (8) Still I dont really understand the difference between Shapley and SHAP ...
4
votes
0answers
61 views

Beneficial dimension for 2nd order modelling in SGD optimization?

There are currently mostly used first order methods in SGD optimizers, second order are often seen too costly as e.g. full Hessian has size $D^2$ in dimension $D$. But we don't need full Hessian - ...
4
votes
1answer
91 views

Why do we use the average-pooling layers instead of max-pooling layers in the DenseNet?

In the paper of DenseNet. The author adopted average-pooling in the transition layers. So what is the motivation of such choice? Why not using max-pooling layers?
4
votes
0answers
64 views

How to reconstruct an image from a training set?

Description: I have taken a series of images/photos of a panorama from different positions (x,y) in space pretty close to each other (max 100m difference). Here there is a top view representation to ...
4
votes
0answers
811 views

BatchNorm after ReLU

I am currently experimenting with different settings for a U-Net (https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) based image segmentation and I was unable to find out if it makes any ...
4
votes
0answers
31 views

Why is it difficult to learn a single kernel that performs well at all positions in the convolutional feature map?

I am reading Deep Learning book by Ian Goodfellow, in which they wrote (in chapter 9, section 9.5) that: " ... MATLAB refers to this as full convolution, in which enough zeros are added for every ...
4
votes
0answers
1k views

Why does Nesterov momentum not improve the rate of convergence in the stochastic gradient case?

I have been reading Deep Learning book by Ian Goodfellow, where they wrote in chapter 8 (section 8.3.3) that Nesterov momentum does not improve the rate of convergence in stochastic gradient case. ...

1
2 3 4 5
52