Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

2,287 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
16
votes
0answers
4k views

Backpropagation on a convolutional layer

Online tutorials describe in depth the convolution of an image with a filter, etc; However, I have not seen one that describes the backpropagation on the filter (at least visually). First let me try ...
10
votes
1answer
2k views

What loss function should I use to score a seq2seq RNN model?

I'm working through the Cho 2014 paper which introduced encoder-decoder architecture for seq2seq modeling. In the paper, they seem to use the probability of the output given input (or it's negative-...
8
votes
0answers
3k views

When should I use the Normal distribution or the Uniform distribution when using Xavier initialization?

Xavier initialization seems to be used quite widely now to initialize connection weights in neural networks, especially deep ones (see What are good initial weights in a neural network?). The ...
7
votes
0answers
73 views

Is there a ML or DL tool that can learn to detect periodically occurring patterns in a one dimensional time series?

I am trying to create a tool that labels refrigerator temperature readings. A reading is taken every 5 minutes, and its label identifies whether of not it was taken while the refrigerator was ...
7
votes
1answer
2k views

Why do people use Zero-Padding in Convolutional Neural Networks?

I am wondering why people usually pad with zeros instead of e.g., using the min-value. Zero-padding, in my opinion, makes sense if you have input images with a pixel range [0, 255] or [0, 1] (after ...
6
votes
0answers
6k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
6
votes
0answers
593 views

How is the loss calculated in a Wasserstein GAN?

I'm trying to implement a Wasserstein GAN according to this blog post: https://myurasov.github.io/2017/09/24/wasserstein-gan-keras.html And it has a wasserstein loss of: ...
6
votes
0answers
626 views

Compatible Function Approximation Theorem in Reinforcement Learning

In the Compatible Function Approximation Theorem, the following condition is required to make the policy gradient to be exact $\nabla J(\theta) = \mathbb{E}_{\pi_{\theta}}\left [\nabla_{\theta}log\pi_{...
6
votes
2answers
2k views

Calculating test-time perplexity for seq2seq (RNN) language models

To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$. My question ...
6
votes
0answers
477 views

Deep Learning vs Structured Learning

I am interested in the differences between using large, deep learning networks vs Probabilistic graphical models (PGMs), like Random Field models, for structured learning (e.g. on images, or labels of ...
5
votes
0answers
294 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
5
votes
0answers
1k views

Validation loss increases while Training loss decrease

I am training a model and the accuracy increases in both the training and validation sets. I am using a pre-trained model as my dataset is very small. I am not sure why the loss increases in the ...
5
votes
1answer
2k views

How to train an LSTM when the sequence has imbalanced classes

I'm labelling sequences at every time step, but some labels in the dataset only occur very briefly between two much more common labels. As a result, the NN is biased towards these common labels. I can'...
5
votes
0answers
78 views

Would gradient boosting machines benefit from adaptive learning rates?

In deep learning, a big deal is made about optimizing an adaptive learning rate. There are numerous popular adaptive learning rate algorithms. The hyperparameters for all of the leading gradient ...
5
votes
0answers
774 views

Why word embeddings learned from word2vec are linearly correlated

I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
5
votes
0answers
594 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
5
votes
1answer
521 views

Does RMSProp/Adam solve vanishing gradient problem?

RMSProp and Adam both scale the effectively learning rate by dividing the moving average of past gradients (root mean squared). So if the first layer has gradient much smaller than the last layer, ...
5
votes
0answers
524 views

Is google's wavenet architecture computing a bunch of values that it will never use?

I've been trying to understand the wavenet paper. In order to do so, I am using this implementation that I found on github because it gets good results and it is pretty clear. But I have a question ...
5
votes
1answer
2k views

Using Rolling Forecast Origin Resampling in R for Neural Network Time Series

I am new to time series prediction and forecasting with neural networks and am having trouble with cross validation. I am fitting a multivariate time series. I have 236 monthly observations. I am ...
5
votes
1answer
79 views

Does $L_1$ regularization help ameliorate the credit assignment problem in (deep) neural nets?

Caveat: this is just a thought that occurred to my while I was driving to work, so maybe it's not well-considered. One challenge to deep neural networks is the credit assignment problem: a node at ...
5
votes
0answers
525 views

TensorFlow Deep MNIST for Experts tutorial: kernels seem to never learn anything

I'm following Google's TensorFlow Deep MNIST for Experts tutorial. Here is my code: http://pastebin.com/ePktssrn The networks seems to get close to 100% accuracy after about training 1000 steps, ...
5
votes
0answers
180 views

How to fix this implementation of Bayesian regularization for ANNs?

I have implemented the Levenberg-Marquard algorithm (from Hagan's "Artifical Neural Network Design" -- 2014) for a two layer network with 20 neurons in the hidden layer. This network can beautifully ...
5
votes
1answer
383 views

Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?

I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font: Our key insight is to exploit the ...
5
votes
0answers
1k views

Recurrent neural network for object tracking & position filtering?

Would a recurrent neural network be appropriate for object tracking tasks? Mainly I will have 3D feature vectors $(x, y, t)$ where $x$ and $y$ are the positions of an object in the image and $t$ is ...
5
votes
0answers
339 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
5
votes
0answers
361 views

What enforces features diversity in RBM?

I'm working on an implementation of a Restricted Boltzman Machine (RBM). I made some tests on the MNIST dataset trying to learn a representation of the digit 2. My inputs are binary images. My aim is ...
5
votes
0answers
425 views

Any implementations of fully recurrent neural networks applied to reinforcement learning?

I've seen a single paper on the topic of adapting fully recurrent networks to a reinforcement learning setting, but according to google scholar its had no citations and no code has been released ...
4
votes
0answers
383 views

Advantage of RMSProp over Adam?

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of ...
4
votes
0answers
58 views

How to reconstruct an image from a training set?

Description: I have taken a series of images/photos of a panorama from different positions (x,y) in space pretty close to each other (max 100m difference). Here there is a top view representation to ...
4
votes
0answers
488 views

How to evaluate performance of (variational) autoencoders?

Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly ...
4
votes
0answers
243 views

what is the mistake of convergence proof in Adam

Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
4
votes
0answers
746 views

BatchNorm after ReLU

I am currently experimenting with different settings for a U-Net (https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) based image segmentation and I was unable to find out if it makes any ...
4
votes
0answers
52 views

Choosing the number of hidden layers and nodes in a Deep Belief Network

What are the recent advances and current best practices in choosing the number and size of stacked Restricted Boltzmann Machines in Deep Belief Networks ?
4
votes
1answer
270 views

Ridge regression: penalizing weights corresponding to larger-scale features

In this article the author is looking at dropout training and trying to show it is equivalent in some way to adding a penalty term to the loss function. On page 5, in the little section called "...
4
votes
1answer
1k views

Are there networks specialised on object detection for a single class of object?

I want to detect the location of a single class of object, which might occur multiple times in an image. Specifically, this relates to research on detecting brake lights for autonomous vehicles. I ...
4
votes
1answer
37 views

How do these matrices form an order-$4$-tensor?

I'm reading this paper on a convolutional neural network for modelling sentences, and I'm having some trouble understanding section $3.5$. Please consider the following text: We denote a feature map ...
4
votes
0answers
147 views

Complex vs. Standard Neural Nets for Complex Data

I've seen some recent papers describing complex valued neural networks like this one. What I'm wondering is, rather than invent a new complex network architecture that takes a complex value as a ...
4
votes
0answers
382 views

Can we apply a constraint on the distribution of the layer output?

As far I understood, the hidden layer outputs can be anything based on the learning algorithm or optimization rules. I was wondering if it possible to some constraints on the layer output. For ...
4
votes
0answers
5k views

Help Understanding Reconstruction Loss In Variational Autoencoder

The reconstruction loss for a VAE (see, for example equation 20.77 in The Deep Learning Book) is often written as $-\mathbb{E}_{z\sim{q(z | x)}} log(p_{model}(x | z))$, where $z$ represents latent ...
4
votes
0answers
253 views

Adversarial examples - regularization method

In Intriguing properties of neural networks (https://arxiv.org/pdf/1312.6199.pdf) they show (4.3), that the existance of adversarial examples is closely connected to the upper Lipschitz constant, ...
4
votes
0answers
905 views

Improving multi-label loss function

Trying to train a CNN on a multilabel problem, each image can have 0, 1, 2 or 3 labels assigned to it. The number of labels is not known a priori. I figured the standard loss function for such problem ...
4
votes
0answers
3k views

Transfer learning on faster rcnn and tensorflow

I am trying to do transfer learning to reuse a pretrained neural net. I got the tensorflow faster rcnn official example to work, and now i would like to reuse it to detect my own classes. This is the ...
4
votes
1answer
813 views

Trouble training LSTM for sequence to sequence learning of sensor time series

I'm experimenting with using RNNs/LSTMs in place of a Kalman Filter (KF) for sensor fusion. I'm struggling to make much progress, and would appreciate some feedback/advice. I have several multi-...
4
votes
1answer
2k views

Condition for RNN vanishing gradients and eigenvalues of the matrix of weights

In this article on recurrent neural networks by Razvan Pascanu, $\mathbf x_t$ is the state at time $t;$ $\mathbf u_t$ the input at time $t$; and $\mathcal E$ is the cost function: A proof is given of ...

1
2 3 4 5
46