366

The batch size defines the number of samples that will be propagated through the network. For instance, let's say you have 1050 training samples and you want to set up a batch_size equal to 100. The algorithm takes the first 100 samples (from 1st to 100th) from the training dataset and trains the network. Next, it takes the second 100 samples (from 101st to ...


364

Verify that your code is bug free There's a saying among writers that "All writing is re-writing" -- that is, the greater part of writing is revising. For programmers (or at least data scientists) the expression could be re-phrased as "All coding is debugging." Any time you're writing code, you need to verify that it works as intended. ...


300

From Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/abs/1609.04836 : The stochastic gradient descent method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a ...


247

Autoencoder is a simple 3-layer neural network where output units are directly connected back to input units. E.g. in a network like this: output[i] has edge back to input[i] for every i. Typically, number of hidden units is much less then number of visible (input/output) ones. As a result, when you pass data through such a network, it first compresses (...


218

In the neural network terminology: one epoch = one forward pass and one backward pass of all the training examples batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need. number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one ...


212

Let's start with a triviliaty: Deep neural network is simply a feedforward network with many hidden layers. This is more or less all there is to say about the definition. Neural networks can be recurrent or feedforward; feedforward ones do not have any loops in their graph and can be organized in layers. If there are "many" layers, then we say that the ...


183

@doug's answer has worked for me. There's one additional rule of thumb that helps for supervised learning problems. You can usually prevent over-fitting if you keep your number of neurons below: $$N_h = \frac{N_s} {(\alpha * (N_i + N_o))}$$ $N_i$ = number of input neurons. $N_o$ = number of output neurons. $N_s$ = number of samples in training data set. $\...


175

Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is $h = \max(0, a)$ where $a = Wx + b$. One major benefit is the reduced likelihood of the gradient to vanish. This arises when $a > 0$. In this regime the gradient has a constant value. In contrast, the gradient ...


170

I'll start making a list here of the ones I've learned so far. As @marcodena said, pros and cons are more difficult because it's mostly just heuristics learned from trying these things, but I figure at least having a list of what they are can't hurt. First, I'll define notation explicitly so there is no confusion: Notation This notation is from Neilsen's ...


169

Suppose that I have a conv layer which outputs an $(N, F, H, W)$ shaped tensor where: $N$ is the batch size $F$ is the number of convolutional filters $H, W$ are the spatial dimensions Suppose the input is fed into a conv layer with $F_1$ 1x1 filters, zero padding and stride 1. Then the output of this 1x1 conv layer will have shape $(N, F_1, H , W)$. So ...


163

One possibility: If you are using dropout regularization layer in your network, it is reasonable that the validation error is smaller than training error. Because usually dropout is activated when training but deactivated when evaluating on the validation set. You get a more smooth (usually means better) function in the latter case.


141

The key/value/query formulation of attention is from the paper Attention Is All You Need. How should one understand the queries, keys, and values The key/value/query concept is analogous to retrieval systems. For example, when you search for videos on Youtube, the search engine will map your query (text in the search bar) against a set of keys (video title,...


137

As a disclaimer, I work on neural nets in my research, but I generally use relatively small, shallow neural nets rather than the really deep networks at the cutting edge of research you cite in your question. I am not an expert on the quirks and peculiarities of very deep networks and I will defer to someone who is. First, in principle, there is no reason ...


128

The original meaning of “Ablation” is the surgical removal of body tissue. The term “Ablation study” has its roots in the field of experimental neuropsychology of the 1960s and 1970s, where parts of animals’ brains were removed to study the effect that this had on their behaviour. In the context of machine learning, and especially complex deep neural ...


108

Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al. There are two reasons for that choice (assuming you have normalized your data, and this is very important): Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of ...


105

It is difficult to be certain without knowing your actual methodology (e.g. cross-validation method, performance metric, data splitting method, etc.). Generally speaking though, training error will almost always underestimate your validation error. However it is possible for the validation error to be less than the training. You can think of it two ways: ...


102

I caution against expecting strong resemblance between biological and artificial neural networks. I think the name "neural networks" is a bit dangerous, because it tricks people into expecting that neurological processes and machine learning should be the same. The differences between biological and artificial neural networks outweigh the similarities. As ...


101

Here are those I understand so far. Most of these work best when given values between 0 and 1. Quadratic cost Also known as mean squared error, this is defined as: $$C_{MST}(W, B, S^r, E^r) = 0.5\sum\limits_j (a^L_j - E^r_j)^2$$ The gradient of this cost function with respect to the output of a neural network and some sample $r$ is: $$\nabla_a C_{MST} = (a^L ...


91

[...] where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the mode. First, you make a prediction using the CNN and obtain the predicted class multinomial distribution ($\sum p_{class} = 1$). Now, in the case of the top-1 score, you check if the top class (the one with ...


91

Global max pooling = ordinary max pooling layer with pool size equals to the size of the input (minus filter size + 1, to be precise). You can see that MaxPooling1D takes a pool_length argument, whereas GlobalMaxPooling1D does not. For example, if the input of the max pooling layer is $0,1,2,2,5,1,2$, global max pooling outputs $5$, whereas ordinary max ...


90

Advantage: Sigmoid: not blowing up activation Relu : not vanishing gradient Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick max(0,$x$) and not perform expensive exponential operations as in Sigmoids Relu : In practice, networks with Relu tend to show better convergence ...


87

The first two algorithms you mention (Nelder-Mead and Simulated Annealing) are generally considered pretty much obsolete in optimization circles, as there are much better alternatives which are both more reliable and less costly. Genetic algorithms covers a wide range, and some of these can be reasonable. However, in the broader class of derivative-free ...


85

The tf.train.AdamOptimizer uses Kingma and Ba's Adam algorithm to control the learning rate. Adam offers several advantages over the simple tf.train.GradientDescentOptimizer. Foremost is that it uses moving averages of the parameters (momentum); Bengio discusses the reasons for why this is beneficial in Section 3.1.1 of this paper. Simply put, this enables ...


84

First of all a general suggestion: do a literature search before you start making experiments on a topic you're not familiar with. You'll save yourself a lot of time. In this case, looking at existing papers you may have noticed that CNNs have been used multiple times for regression: this is a classic but it's old (yes, 3 years is old in DL). A more ...


82

Relation to Word2Vec ========================================== Word2Vec in a simple picture: More in-depth explanation: I believe it's related to the recent Word2Vec innovation in natural language processing. Roughly, Word2Vec means our vocabulary is discrete and we will learn an map which will embed each word into a continuous vector space. Using this ...


79

From Introduction to Neural Networks for Java (second edition) by Jeff Heaton - preview freely available at Google Books and previously at author's website: The Number of Hidden Layers There are really two decisions that must be made regarding the hidden layers: how many hidden layers to actually have in the neural network and how many neurons will ...


79

The negative log likelihood (eq.80) is also known as the multiclass cross-entropy (ref: Pattern Recognition and Machine Learning Section 4.3.4), as they are in fact two different interpretations of the same formula. eq.57 is the negative log likelihood of the Bernoulli distribution, whereas eq.80 is the negative log likelihood of the multinomial ...


78

The "sample size" you're talking about is referred to as batch size, $B$. The batch size parameter is just one of the hyper-parameters you'll be tuning when you train a neural network with mini-batch Stochastic Gradient Descent (SGD) and is data dependent. The most basic method of hyper-parameter search is to do a grid search over the learning rate and batch ...


77

A recent paper The Loss Surfaces of Multilayer Networks offers some possible explanations for this. From their abstract (bold is mine): "We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This ...


76

The posted answers are great, and I wanted to add a few "Sanity Checks" which have greatly helped me in the past. 1) Train your model on a single data point. If this works, train it on two inputs with different outputs. This verifies a few things. First, it quickly shows you that your model is able to learn by checking if your model can overfit your data. ...


Only top voted, non community-wiki answers of a minimum length are eligible