Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

2,665 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
12
votes
0answers
6k views

When should I use the Normal distribution or the Uniform distribution when using Xavier initialization?

Xavier initialization seems to be used quite widely now to initialize connection weights in neural networks, especially deep ones (see What are good initial weights in a neural network?). The ...
11
votes
0answers
4k views

Validation loss increases while Training loss decrease

I am training a model and the accuracy increases in both the training and validation sets. I am using a pre-trained model as my dataset is very small. I am not sure why the loss increases in the ...
8
votes
1answer
97 views

Is there a ML or DL tool that can learn to detect periodically occurring patterns in a one dimensional time series?

I am trying to create a tool that labels refrigerator temperature readings. A reading is taken every 5 minutes, and its label identifies whether of not it was taken while the refrigerator was ...
8
votes
0answers
830 views

How to evaluate performance of (variational) autoencoders?

Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly ...
7
votes
0answers
38 views

What's up with Neural Stochastic Differential Equations from a practical standpoint?

I've spent a few days reading some of the new papers about Neural SDEs. For example, here is one from Tzen and Raginsky and here is one that came out simultaneously by Peluchetti and Favaro. There are ...
7
votes
0answers
496 views

what is the mistake of convergence proof in Adam

Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
7
votes
1answer
526 views

Deep Learning vs Structured Learning

I am interested in the differences between using large, deep learning networks vs Probabilistic graphical models (PGMs), like Random Field models, for structured learning (e.g. on images, or labels of ...
6
votes
0answers
372 views

What does it mean to take the expectation with respect to a probability distribution?

I see this expectation in a lot of machine learning literature: $$\mathbb{E}_{p(\mathbf{x};\mathbf{\theta})}[f(\mathbf{x};\mathbf{\phi})] = \int p(\mathbf{x};\mathbf{\theta}) f(\mathbf{x};\mathbf{\phi}...
6
votes
1answer
476 views

Clarification: Are Generative Adversarial Networks an alternative to MCMC sampling?

I have been reading the original Goodfellow, et. al. paper on Generative Adversarial Networks and the way that they can obtain estimates of the posterior distribution of a discriminative network or ...
6
votes
0answers
79 views

Choosing the number of hidden layers and nodes in a Deep Belief Network

What are the recent advances and current best practices in choosing the number and size of stacked Restricted Boltzmann Machines in Deep Belief Networks ?
6
votes
0answers
7k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
6
votes
0answers
664 views

How is the loss calculated in a Wasserstein GAN?

I'm trying to implement a Wasserstein GAN according to this blog post: https://myurasov.github.io/2017/09/24/wasserstein-gan-keras.html And it has a wasserstein loss of: ...
6
votes
0answers
803 views

Compatible Function Approximation Theorem in Reinforcement Learning

In the Compatible Function Approximation Theorem, the following condition is required to make the policy gradient to be exact $\nabla J(\theta) = \mathbb{E}_{\pi_{\theta}}\left [\nabla_{\theta}log\pi_{...
6
votes
0answers
436 views

Any implementations of fully recurrent neural networks applied to reinforcement learning?

I've seen a single paper on the topic of adapting fully recurrent networks to a reinforcement learning setting, but according to google scholar its had no citations and no code has been released ...
5
votes
0answers
49 views

Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
5
votes
0answers
350 views

Understanding Object2Vec

AWS released an interesting feature as part of the SageMaker service called Object2Vec that lets you make an embedding for search out of pretty much anything: documents, users, products, ...
5
votes
0answers
388 views

Difference between Shapley values and SHAP

The Paper regarding die SHAP value gives a formula for the Shapley Values in (4) and for SHAP values apparently (?) in (8) Still I dont really understand the difference between Shapley and SHAP ...
5
votes
0answers
448 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
5
votes
0answers
1k views

How to normalize input data for autoencoders - anomaly detection

I'm building an autoencoder to identify anomalies on numerical data. The input features have different scales (i.e. some take values from 0 to 5, while others can be much much higher) and most of them ...
5
votes
1answer
463 views

Ridge regression: penalizing weights corresponding to larger-scale features

In this article the author is looking at dropout training and trying to show it is equivalent in some way to adding a penalty term to the loss function. On page 5, in the little section called "...
5
votes
0answers
122 views

Would gradient boosting machines benefit from adaptive learning rates?

In deep learning, a big deal is made about optimizing an adaptive learning rate. There are numerous popular adaptive learning rate algorithms. The hyperparameters for all of the leading gradient ...
5
votes
0answers
940 views

Why word embeddings learned from word2vec are linearly correlated

I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
5
votes
0answers
724 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
5
votes
0answers
466 views

Can we apply a constraint on the distribution of the layer output?

As far I understood, the hidden layer outputs can be anything based on the learning algorithm or optimization rules. I was wondering if it possible to some constraints on the layer output. For ...
5
votes
0answers
610 views

Is google's wavenet architecture computing a bunch of values that it will never use?

I've been trying to understand the wavenet paper. In order to do so, I am using this implementation that I found on github because it gets good results and it is pretty clear. But I have a question ...
5
votes
2answers
870 views

First-layer Visualizations in a neural network

I am reading the lectures on "Convolutional Neural Networks for Visual Recognition", and in this lecture they deal with first layer visualization. As you can see in the figure below- this figure ...
5
votes
0answers
893 views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
5
votes
1answer
2k views

Using Rolling Forecast Origin Resampling in R for Neural Network Time Series

I am new to time series prediction and forecasting with neural networks and am having trouble with cross validation. I am fitting a multivariate time series. I have 236 monthly observations. I am ...
5
votes
1answer
84 views

Does $L_1$ regularization help ameliorate the credit assignment problem in (deep) neural nets?

Caveat: this is just a thought that occurred to my while I was driving to work, so maybe it's not well-considered. One challenge to deep neural networks is the credit assignment problem: a node at ...
5
votes
0answers
1k views

Training batch size in relation to number of classes in a neural network

I'm using Keras on top of Theano for neural network training. What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't ...
5
votes
0answers
526 views

TensorFlow Deep MNIST for Experts tutorial: kernels seem to never learn anything

I'm following Google's TensorFlow Deep MNIST for Experts tutorial. Here is my code: http://pastebin.com/ePktssrn The networks seems to get close to 100% accuracy after about training 1000 steps, ...
5
votes
0answers
191 views

How to fix this implementation of Bayesian regularization for ANNs?

I have implemented the Levenberg-Marquard algorithm (from Hagan's "Artifical Neural Network Design" -- 2014) for a two layer network with 20 neurons in the hidden layer. This network can ...
5
votes
1answer
393 views

Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?

I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font: Our key insight is to exploit the ...
5
votes
0answers
1k views

Recurrent neural network for object tracking & position filtering?

Would a recurrent neural network be appropriate for object tracking tasks? Mainly I will have 3D feature vectors $(x, y, t)$ where $x$ and $y$ are the positions of an object in the image and $t$ is ...
5
votes
0answers
362 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
5
votes
1answer
2k views

CNN: Details of Zeiler Fergus Net

I want to replicate the modified AlexNet by Zeiler and Fergus from 2013 (Visualizing and Understanding Convolutional Networks) but they spare some details. Hope someone here knows more about it. What ...
5
votes
1answer
148 views

Is there any procedure to determine the number of layers of convolution and pooling needs in CNN?

When I want to use Caffe to create my own CNN, how to determine the number of convolution and pooling layer I need is suitable to extract correct features basis? Is there any principal or documents? ...
5
votes
0answers
366 views

What enforces features diversity in RBM?

I'm working on an implementation of a Restricted Boltzman Machine (RBM). I made some tests on the MNIST dataset trying to learn a representation of the digit 2. My inputs are binary images. My aim is ...
5
votes
1answer
841 views

Bayesian hyperparameter optimization + cross-validation

I want to use Bayesian optimization to search a space of hyperparameters for a neural network model. My objective function for this optimization is validation set accuracy. In addition, I want to ...
4
votes
1answer
673 views

Does Attention Help with standard auto-encoders

I understand the use of attention mechanisms in the encoder-decoder for sequence-to-sequence problem such as a language translator. I am just trying to figure out whether it is possible to use ...
4
votes
1answer
242 views

Why are the tied weights in autoencoders transposed and not inverted?

I am currently reading about Autoencoders. From what I understand so far, when we are dealing with a symmetrical autoencoder, a good practice is to tie the weights of the decoder layers to the weights ...
4
votes
1answer
62 views

Does gradient descent assume updates of one layer/parameter at a time?

I read the following in "Deep Learning", from Goodfellow et al (Chapter 8, page 313): The gradient tells how to update each parameter, under the assumption that the other layers do not change. In ...
4
votes
0answers
48 views

Why is homoscedasticity (homogeneity of variance) important in neural network layers?

I'm studying the famous Xavier initialization paper (Understanding the Difficulty of Training Deep Feedforward Neural Networks (Glorot and Bengio, 2010)) and had a question. When they explain the ...
4
votes
0answers
38 views

What assumptions about time series data are neccessary to use a stateless LSTM?

Basically I am trying to model a time series using an LSTM layer, and I was wondering whether I should be using a stateful or stateless LSTM layer. More specifically I was wondering what assumptions ...
4
votes
0answers
2k views

Advantage of RMSProp over Adam?

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of ...

1
2 3 4 5
54