Questions tagged [neural-networks]

Artificial neural networks (ANNs) are a broad class of computational models loosely based on biological neural networks. They encompass feedforward NNs (including "deep" NNs), convolutional NNs, recurrent NNs, etc.

2,883 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
18 votes
0 answers
11k views

When should I use the Normal distribution or the Uniform distribution when using Xavier initialization?

Xavier initialization seems to be used quite widely now to initialize connection weights in neural networks, especially deep ones (see What are good initial weights in a neural network?). The ...
user avatar
  • 2,020
12 votes
0 answers
762 views

what is the mistake of convergence proof in Adam

Sashank J. Reddi et. al in their paper "On the convergence of Adam and beyond" say that, Adam's proof of convergence as stated in original paper is wrong. More than that, they point out that the value ...
user avatar
10 votes
0 answers
2k views

What does it mean to take the expectation with respect to a probability distribution?

I see this expectation in a lot of machine learning literature: $$\mathbb{E}_{p(\mathbf{x};\mathbf{\theta})}[f(\mathbf{x};\mathbf{\phi})] = \int p(\mathbf{x};\mathbf{\theta}) f(\mathbf{x};\mathbf{\phi}...
user avatar
  • 3,026
10 votes
0 answers
1k views

How to evaluate performance of (variational) autoencoders?

Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly ...
user avatar
  • 303
9 votes
1 answer
118 views

Is there a ML or DL tool that can learn to detect periodically occurring patterns in a one dimensional time series?

I am trying to create a tool that labels refrigerator temperature readings. A reading is taken every 5 minutes, and its label identifies whether of not it was taken while the refrigerator was ...
user avatar
8 votes
2 answers
937 views

Why are the embeddings of tokens multiplied by $\sqrt D$ (note not divided by square root of D) in a transformer?

Why does the transformer tutorial in PyTorch have a multiplication by sqrt number of inputs? I know there is a division by sqrt(D) in the multiheaded self attention, but why is there something similar ...
user avatar
8 votes
0 answers
71 views

What's up with Neural Stochastic Differential Equations from a practical standpoint?

I've spent a few days reading some of the new papers about Neural SDEs. For example, here is one from Tzen and Raginsky and here is one that came out simultaneously by Peluchetti and Favaro. There are ...
user avatar
8 votes
0 answers
907 views

Difference between Shapley values and SHAP

The Paper regarding die SHAP value gives a formula for the Shapley Values in (4) and for SHAP values apparently (?) in (8) Still I dont really understand the difference between Shapley and SHAP ...
user avatar
  • 203
8 votes
0 answers
895 views

Compatible Function Approximation Theorem in Reinforcement Learning

In the Compatible Function Approximation Theorem, the following condition is required to make the policy gradient to be exact $\nabla J(\theta) = \mathbb{E}_{\pi_{\theta}}\left [\nabla_{\theta}log\pi_{...
user avatar
7 votes
0 answers
5k views

Advantage of RMSProp over Adam?

I've learned from DL classes that Adam should be the default choice for neural network training. However, I've recently seen more and more recent reinforcement learning agents use RMSProp instead of ...
user avatar
  • 915
7 votes
0 answers
189 views

Would gradient boosting machines benefit from adaptive learning rates?

In deep learning, a big deal is made about optimizing an adaptive learning rate. There are numerous popular adaptive learning rate algorithms. The hyperparameters for all of the leading gradient ...
user avatar
  • 2,060
7 votes
0 answers
7k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
user avatar
  • 171
7 votes
0 answers
655 views

Is google's wavenet architecture computing a bunch of values that it will never use?

I've been trying to understand the wavenet paper. In order to do so, I am using this implementation that I found on github because it gets good results and it is pretty clear. But I have a question ...
user avatar
7 votes
0 answers
711 views

How is the loss calculated in a Wasserstein GAN?

I'm trying to implement a Wasserstein GAN according to this blog post: https://myurasov.github.io/2017/09/24/wasserstein-gan-keras.html And it has a wasserstein loss of: ...
user avatar
7 votes
0 answers
1k views

ReLU derivative - second order effects

I am reading the Deep Learning Book, where there is a section on generalisations of the ReLU (section 6.3.1). It states: The second derivative of the rectifying operation is 0 almost everywhere, ...
user avatar
  • 541
7 votes
1 answer
566 views

Deep Learning vs Structured Learning

I am interested in the differences between using large, deep learning networks vs Probabilistic graphical models (PGMs), like Random Field models, for structured learning (e.g. on images, or labels of ...
user avatar
  • 1,854
7 votes
0 answers
446 views

Any implementations of fully recurrent neural networks applied to reinforcement learning?

I've seen a single paper on the topic of adapting fully recurrent networks to a reinforcement learning setting, but according to google scholar its had no citations and no code has been released ...
user avatar
  • 407
6 votes
1 answer
130 views

Why convert spectrogram to RGB for machine learning?

I've seen a few publications that feed an RGB image of a spectrogram to a neural net, and someone claiming a network does better with RGB than grayscale. A spectrogram is fundamentally a 2D ...
user avatar
6 votes
1 answer
2k views

Does Attention Help with standard auto-encoders

I understand the use of attention mechanisms in the encoder-decoder for sequence-to-sequence problem such as a language translator. I am just trying to figure out whether it is possible to use ...
user avatar
  • 173
6 votes
0 answers
75 views

Local optima in high-dimensional optimization

I remember a theorem along the lines of In higher dimensional optimization problems, you are less likely to get stuck in local optima, because the more dimensions you have, the more likely you are to ...
user avatar
  • 327
6 votes
0 answers
572 views

Understanding Object2Vec

AWS released an interesting feature as part of the SageMaker service called Object2Vec that lets you make an embedding for search out of pretty much anything: documents, users, products, ...
user avatar
  • 6,047
6 votes
0 answers
291 views

What machine learning and deep learning models are used for longitudinal studies (panel data)?

As the title suggests, I have a longitudinal database (also called panel data). (I have over 100.000 observations. The time period is X years. This means that for every year I have the values of the ...
user avatar
6 votes
0 answers
635 views

Usage of dropout in convolutional GANs with batch norm?

In DCGAN, dropout is not used in either generator or discriminator. When using batch norm, are the benefits of dropout generally so marginal that is is not used? If it is used, in what circumstances?...
user avatar
  • 2,271
6 votes
0 answers
2k views

Multi-target Regression Neural Network: Trade Off

Suppose you have a number of input features, for example: x1 - temperature x2 - day of the week x3 - quantity of rainfall ... You are trying to predict a number of output targets - using neural ...
user avatar
6 votes
0 answers
97 views

Choosing the number of hidden layers and nodes in a Deep Belief Network

What are the recent advances and current best practices in choosing the number and size of stacked Restricted Boltzmann Machines in Deep Belief Networks ?
user avatar
  • 49.6k
6 votes
1 answer
2k views

Bayesian hyperparameter optimization + cross-validation

I want to use Bayesian optimization to search a space of hyperparameters for a neural network model. My objective function for this optimization is validation set accuracy. In addition, I want to ...
user avatar
  • 61
5 votes
0 answers
2k views

How to normalize input data for autoencoders - anomaly detection

I'm building an autoencoder to identify anomalies on numerical data. The input features have different scales (i.e. some take values from 0 to 5, while others can be much much higher) and most of them ...
user avatar
  • 538
5 votes
0 answers
160 views

Complex vs. Standard Neural Nets for Complex Data

I've seen some recent papers describing complex valued neural networks like this one. What I'm wondering is, rather than invent a new complex network architecture that takes a complex value as a ...
user avatar
  • 693
5 votes
0 answers
1k views

Why word embeddings learned from word2vec are linearly correlated

I was playing with CBOW from the word2vec program downloaded from https://code.google.com/archive/p/word2vec/ with some sequence data (peptides in this case). I was trying to get embeddings for amino ...
user avatar
  • 917
5 votes
0 answers
820 views

Why no one talks about stochastic conjugate gradient descent?

As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the ...
user avatar
  • 151
5 votes
0 answers
2k views

Reinforcement learning with large number of actions to choose from

In reinforcement learning, if the number of possible actions to choose from is increased, it becomes difficult to train. In many successful examples (ex: DQN), the number of possible actions to choose ...
user avatar
5 votes
0 answers
572 views

Can we apply a constraint on the distribution of the layer output?

As far I understood, the hidden layer outputs can be anything based on the learning algorithm or optimization rules. I was wondering if it possible to some constraints on the layer output. For ...
user avatar
5 votes
0 answers
399 views

What is the difference between using Siamese networks and MANN in one shot learning?

Memory Augmented Neural Networks have an external memory which helps to learn something with the low amount of data. It also can manipulate the memory in a dynamic manner. On another hand, siamese ...
user avatar
5 votes
0 answers
1k views

Triplet Deep Learning Embedding Loss Functions

Triplet embeddings consist of mapping a group of images to an embedding space, such that images deemed more similar to each other end up closer together. The "triplet" comes from training, ...
user avatar
  • 13.2k
5 votes
1 answer
2k views

Using Rolling Forecast Origin Resampling in R for Neural Network Time Series

I am new to time series prediction and forecasting with neural networks and am having trouble with cross validation. I am fitting a multivariate time series. I have 236 monthly observations. I am ...
user avatar
5 votes
0 answers
1k views

Training batch size in relation to number of classes in a neural network

I'm using Keras on top of Theano for neural network training. What should be my batch size in relation to the number of classes? I have 560 classes and if I use a batch size more than 128, I can't ...
user avatar
  • 151
5 votes
0 answers
659 views

Normalizing data worsens the performance of CNN?

I've been using CNN for facial recognition tasks, first I train a CNN for classification, and I use the trained CNN to extract features from images and do verification (tell whether two pictures are ...
user avatar
  • 14.1k
5 votes
0 answers
230 views

How to fix this implementation of Bayesian regularization for ANNs?

I have implemented the Levenberg-Marquard algorithm (from Hagan's "Artifical Neural Network Design" -- 2014) for a two layer network with 20 neurons in the hidden layer. This network can ...
user avatar
5 votes
1 answer
413 views

Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?

I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font: Our key insight is to exploit the ...
user avatar
  • 143
5 votes
0 answers
1k views

Recurrent neural network for object tracking & position filtering?

Would a recurrent neural network be appropriate for object tracking tasks? Mainly I will have 3D feature vectors $(x, y, t)$ where $x$ and $y$ are the positions of an object in the image and $t$ is ...
user avatar
  • 2,695
5 votes
0 answers
388 views

Interpreting hidden layer representations in ANNs

I'm using the fann library for writing an Artificial Neural Network in C++. I trained my network for the task of recognizing faces inside a set of 128x128 .png images, using three different algorithms:...
user avatar
  • 151
5 votes
1 answer
2k views

CNN: Details of Zeiler Fergus Net

I want to replicate the modified AlexNet by Zeiler and Fergus from 2013 (Visualizing and Understanding Convolutional Networks) but they spare some details. Hope someone here knows more about it. What ...
user avatar
  • 51
5 votes
0 answers
371 views

What enforces features diversity in RBM?

I'm working on an implementation of a Restricted Boltzman Machine (RBM). I made some tests on the MNIST dataset trying to learn a representation of the digit 2. My inputs are binary images. My aim is ...
user avatar
5 votes
0 answers
873 views

Is NN saturation always bad?

I am trying to analyse the effect of hidden unit saturation (outputting mostly 0 and 1 for sigmoid, and not much in-between) on the neural network training performance, and I feel a bit stuck, theory-...
user avatar
4 votes
0 answers
55 views

What is a "neural network prior" in this context of physics informed neural networks?

In the paper "Physics Informed Deep Learning (Part I): Data-driven solutions of nonlinear partial differential equations" (https://arxiv.org/abs/1711.10561v1), basically this paper uses a ...
user avatar
  • 221

1
2 3 4 5
58