87

The first two algorithms you mention (Nelder-Mead and Simulated Annealing) are generally considered pretty much obsolete in optimization circles, as there are much better alternatives which are both more reliable and less costly. Genetic algorithms covers a wide range, and some of these can be reasonable. However, in the broader class of derivative-free ...


78

The "sample size" you're talking about is referred to as batch size, $B$. The batch size parameter is just one of the hyper-parameters you'll be tuning when you train a neural network with mini-batch Stochastic Gradient Descent (SGD) and is data dependent. The most basic method of hyper-parameter search is to do a grid search over the learning rate and batch ...


63

You have every reason to be confused, because in supervised learning one doesn't need to backpropagate to labels. They are considered fixed ground truth and only the weights need to be adjusted to match them. But in some cases, the labels themselves may come from a differentiable source, another network. One example might be adversarial learning. In this ...


49

$$f=\sum w_ix_i+b$$ $$\frac{df}{dw_i}=x_i$$ $$\frac{dL}{dw_i}=\frac{dL}{df}\frac{df}{dw_i}=\frac{dL}{df}x_i$$ because $x_i>0$, the gradient $\dfrac{dL}{dw_i}$ always has the same sign as $\dfrac{dL}{df}$ (all positive or all negative). Update Say there are two parameters $w_1$ and $w_2$. If the gradients of two dimensions are always of the same sign (i.e.,...


46

Note: I am not an expert on backprop, but now having read a bit, I think the following caveat is appropriate. When reading papers or books on neural nets, it is not uncommon for derivatives to be written using a mix of the standard summation/index notation, matrix notation, and multi-index notation (include a hybrid of the last two for tensor-tensor ...


37

Yan LeCun and others argue in Efficient BackProp that Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme case where all the inputs are positive. Weights to a particular node in the first weight layer are updated by an amount proportional to $\delta x$ where $\delta$ ...


36

Because we can't. The optimization surface $S(\mathbf{w})$ as a function of the weights $\mathbf{w}$ is nonlinear and no closed form solution exists for $\frac{d S(\mathbf{w})}{d\mathbf{w}}=0$. Gradient descent, by definition, descends. If you reach a stationary point after descending, it has to be a (local) minimum or a saddle point, but never a local ...


33

TL;DR: For time series and density modeling, neural ODEs offer some benefits that we don't know how to get otherwise. For plain supervised learning, there are potential computational benefits, but for practical purposes they probably aren't worth using yet in that setting. To answer your first question: Is there something NeuralODEs do that "...


29

I figured I'd answer a self-contained post here for anyone that's interested. This will be using the notation described here. Introduction The idea behind backpropagation is to have a set of "training examples" that we use to train our network. Each of these has a known answer, so we can plug them into the neural network and find how much it was ...


29

Average. Examples: Notes to Andrew Ng's Machine Learning Course on Coursera compiled by Alex Holehouse. Summing the gradients due to individual samples you get a much smoother gradient. The larger the batch the smoother the resulting gradient used in updating the weight. Dividing the sum by the batch size and taking the average gradient has the effect of: ...


26

While @GeoMatt22's answer is correct, I personally found it very useful to reduce the problem to a toy example and draw a picture: I then defined the operations each node was computing, treating the $h$'s and $w$'s as inputs to a "network" ($\mathbf{t}$ is a one-hot vector representing the class label of the data point): $$L=-t_1\log o_1 -t_2\log o_2$$ $$...


24

Gradient descent doesn't quite work the way you suggested but a similar problem can occur. We don't calculate the average loss from the batch, we calculate the average gradients of the loss function. The gradients are the derivative of the loss with respect to the weight and in a neural network the gradient for one weight depends on the inputs of that ...


23

Yes, the neurons are considered zero during backpropagation as well. Otherwise dropout wouldn't do anything! Remember that forward propagation during training is only used to set up the network for backpropagation, where the network is actually modified (as well as for tracking training error and such). In general, it's important to account for anything ...


22

Add sends the gradient back equally to both inputs. You can convince yourself of this by running the following in tensorflow: import tensorflow as tf graph = tf.Graph() with graph.as_default(): x1_tf = tf.Variable(1.5, name='x1') x2_tf = tf.Variable(3.5, name='x2') out_tf = x1_tf + x2_tf grads_tf = tf.gradients(ys=[out_tf], xs=[x1_tf, ...


21

I'm going to answer your question about the $\delta_i^{(l)}$, but remember that your question is a sub question of a larger question which is why: $$\nabla_{ij}^{(l)} = \sum_k \theta_{ki}^{(l+1)}\delta_k^{(l+1)}*(a_i^{(l)}(1-a_i^{(l)})) * a_j^{(l-1)}$$ Reminder about the steps in Neural networks: Step 1: forward propagation (calculation of the $a_{i}^{(l)}$...


21

It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and they also share a similar "trend". Needless to say that the $\tanh$ function is called a shifted version of the $\text{sigmoid}$ function. The real reason that $\...


19

First, let's lay out what we have got and our assumptions about the shapes of different vectors. Let, $|W|$ be the number of words in the vocab $y$ and $\hat{y}$ be column vectors of shape $|W|$ x 1 $u_i$ and $v_j$ be the column vectors of shape $D$ X 1 ($D$ = dimension of embeddings) $y$ be the one-hot encoded column vector of shape $|W|$ x 1 $\hat{y}$ ...


17

Local minima are not really as great a problem with neural nets as is often suggested. Some of the local minima are due to the symmetry of the network (i.e. you can permute the hidden neurons and leave the function of the network unchanged. All that is necessary is to find a good local minima, rather than the global minima. As it happens aggressively ...


17

Here's a paper dedicated to this very question: Parascandolo and Virtanen (2016). Taming the waves: sine as activation function in deep neural networks. Some key points from the paper: Sinusoidal activation functions have been largely ignored, and are considered difficult to train. They review past work that has used sinusoidal activation functions. ...


16

Well, the original neural networks, before the backpropagation revolution in the 70s, were "trained" by hand. :) That being said: There is a "school" of machine learning called extreme learning machine that does not use backpropagation. What they do do is to create a neural network with many, many, many nodes --with random weights-- and then train the ...


15

You are correct that if you try to directly optimize the SVM's accuracy on training cases, also called the 0-1 loss, the gradient vanishes. This is why people don't do that. :) What you're trying to do, though, isn't really an SVM yet; it's rather just a general linear classifier. An SVM in particular arises when you replace the 0-1 loss function with a ...


15

The following assumes a loss function $f$ that's expressed as a sum, not an average. Expressing the loss as an average means that the scaling $\frac{1}{n}$ is "baked in" and no further action is needed. In particular, note that F.mse_loss uses reduction="mean" by default, so in the case of OP's code, no further modification is necessary ...


14

According to Wikipedia, there are 4 main types of artificial neural network learning algorithms: supervised, unsupervised, reinforcement and deep learning. Unsupervised learning algorithms: Perceptron, Self-organizing map, Radial basis function network Supervised learning algorithms: Backpropagation, Autoencoders, Hopfield networks, Boltzmann machines, ...


14

Extending @Dikran Marsupial's answer.... Anna Choromanska and her colleagues in Yan LeCunn's group at NYU, address this in their 2014 AISTATS paper "The Loss Surface of Multilayer Nets". Using random matrix theory, along with some experiments, they argue that: For large-size networks, most local minima are equivalent and yield similar performance on a ...


14

I think its about saturation of the neurons. Think about you have an activation function like sigmoid. If your weight val gets value >= 2 or <=-2 your neuron will not learn. So, if you truncate your normal distribution you will not have this issue(at least from the initialization) based on your variance. I think thats why, its better to use truncated ...


14

There are all sorts of local search algorithms you could use, backpropagation has just proved to be the most efficient for more complex tasks in general; there are circumstances where other local searches are better. You could use random-start hill climbing on a neural network to find an ok solution quickly, but it wouldn't be feasible to find a near ...


14

A very short answer: LSTM decouples cell state (typically denoted by c) and hidden layer/output (typically denoted by h), and only do additive updates to c, which makes memories in c more stable. Thus the gradient flows through c is kept and hard to vanish (therefore the overall gradient is hard to vanish). However, other paths may cause gradient explosion. ...


14

The reparameterization trick. $$x = \text{sample}(\mathcal{N}(\mu, \sigma^2))$$ is not backpropable wrt $\mu$ or $\sigma$. However, we can rewrite this as: $$x = \mu + \sigma\ \text{sample}( \mathcal{N}(0, 1))$$ which is clearly equivalent and backpropable.


13

It's true that limiting your gradient propagation to 30 time steps will prevent it from learning everything possible in your dataset. However, it depends strongly on your dataset whether that will prevent it from learning important things about the features in your model! Limiting the gradient during training is more like limiting the window over which your ...


13

The mae, as a function of $y_{\text{pred}}$, is not differentiable at $y_{\text{pred}}=y_{\text{true}}$. Elsewhere, the derivative is $\pm 1$ by a straightforward application of the chain rule: $$\dfrac{d\text{MAE}}{dy_{\text{pred}}} = \begin{cases} +1,\quad y_{\text{pred}}>y_{\text{true}}\\ -1,\quad y_{\text{pred}}<y_{\text{true}} \end{cases}$$ ...


Only top voted, non community-wiki answers of a minimum length are eligible