New answers tagged

1

It appears this is due to the instability of the hidden features and the dynamics of gradient descent with these shifting features. I did some experiments fitting with a small network with 3 hidden neurons, fitting one dimensional nonlinear data. If you freeze the first layer, and only run gradient descent on the last layer, then the residuals go to zero: ...


3

Triplet models are notoriously tricky to train. Before starting a triplet loss project, I strongly recommend reading "FaceNet: A Unified Embedding for Face Recognition and Clustering" by Florian Schroff, Dmitry Kalenichenko, James Philbin because it outlines some of the key problems that arise when using triplet losses, as well as suggested ...


0

If your input has mean zero and standard deviation one, and your hidden layers are properly initialized then the output of your very first forward pass should have mean zero and standard deviation one (as I understand it). If the activation function on your output is softmax then your output for three classes will be [0.333, 0.333, 0.333] while the targets ...


0

Generally no this isn't done, since NMS isn't a very differentiable operation by it's discrete nature. There has been work on Learning NMS, which replaces NMS with a backpropable trained component.


1

I see two issues. Accuracy is a surprisingly bad performance metric. If you evaluate your model using a so-called proper scoring rule like cross-entropy loss or Brier score, you may find that the out-of-sample performance improves even though accuracy decreases. This is because accuracy relies on a threshold, and the threshold that gives the beat accuracy ...


0

I think the problem here is one of notation. The notation $X \sim p_G$ as used in the GAN paper is not meant to indicate $X$ is distributed according to $p_G$, but rather sampled from $p_G$. Of course this raises the problem about why the following equality holds in the proof: $$ \mathbb{E}_{Z \sim p_Z}\left[ \log (1-D(G(z))\right] = \mathbb{E}_{X \sim p_G}\...


1

Exponentials of very small numbers can under flow to 0. But this will never happen if you work on the logit scale. So, use logits. The algebra is tedious but you can rewrite cross entropy loss with softmax/sigmoid loss as an expression of logits. Elements of Statistical Learning does this in its discussion of logistic regression (section 4.4.1, p. 120). ...


3

There is no simple answer to such question. You can train neural network with one sample, you'd just overfit to it. Moreover, there are some recent results that in some cases neural networks with few orders of magnitude more parameters then samples can achieve better test set performance then smaller networks. Such rules of thumb don’t even work for much ...


1

In the first one, there is data leakage between training and test sets. The preprocessing should take place in the training set, and using the fitted preprocessing modules, you should transform your test and validation data. So, the first one is expected to be more optimistic on the performance. In addition, as far as I understand, you don't need time series ...


1

Yes they're the same. The 1x1 convolution is in both of them. For residual mappings, you're adding the old layer's input value to the input of the later layer down the line (aka up the image). If the convolutional layers in between decrease the dimensionality of the image (for example by using zero padding), then the input dimension will be smaller for the ...


2

On the "overlap-tile strategy" specifically: The blue box in Fig 2 (left) shows the input to the network. Because they're using valid convolutions, the output is the smaller yellow box (right). Sounds like you understand this part already. They're trying to show that the image that they want to predict on is bigger than the input to the network (...


0

What architecture is your neural network? Classifying on raw pixels is hard, even for greyscale images of that size. If you're not using a Convolutional neural network or other architecture designed for image classification, the task might be too difficult for your network. Learning from fewer features is a lot easier and faster. You may need to train longer,...


1

RNN's operate on the input sequence one at a time going down the line. Transformers have input width greater than the length of the longest input sequence. It eats up the whole sequence at once, chews it through the different attention layers, then spits it out. So it can attend to anywhere in the input at any time, but this means you can't run a given model ...


0

From the paper: Most ODE solvers have the option to output the state $\mathbf{z}(t)$ at multiple times. When the loss depends on these intermediate states, the reverse-mode derivative must be broken into a sequence of separate solves, one between each consecutive pair of output times (Figure 2). At each observation, the adjoint must be adjusted in the ...


0

You can test and train at the same time so long as you test on a data point first (forward validation) and then use that data point to train. To understand this compared with testing separately from training, consider the batch size with which you're testing and training. If your batch size is large, then it's more like you're testing and training ...


0

Here is the thing, I think the authors use the notation in eqn.4 in reverse order i.e, $z_L \rightarrow z_{L-1} ... \rightarrow z_1 \rightarrow x$. In this notation $z_L$ has a normal distribution $N(0,I)$ and x is the data distribution whose likelihood is supposed to be maximized. Section 3.1 provides background on the theory of transforming random ...


2

I'll tell you what I know/have heard about it. There's probably other analysis out there I haven't seen. I think multi-head attention was introduced in the transformer network paper. Their justification/explanation is: Another paper on a modified Transformer architecture questions this understanding though:


1

Yeah their description is a bit confusing. I agree with your interpretation. This paper has 15k citations, so if that strategy is effective, it's probably pretty commonly used. Otherwise, my only guess could be that other techniques in the paper were more important. I'm not familiar with biomedical computer vision research though.


0

You may be suffering from a common issue of neural networks failing to generalize to numerical inputs unseen in training. The best display of this behavior I know is the figure from this paper: Caption from the paper: MLPs learn the identity function only for the range of values they are trained on. The mean error ramps up severely both below and above the ...


1

If you're given a lot of freedom in the algorithm design, you can do the following : train one huge but shallow (ad probably non-convolutional, you really want it very powerful but very stupid) neural network to memorize the training set perfectly, as suggested by @Peteris and @Wololo (his solution has converted me). This network should give you both the ...


1

It depends. One way to think about this problem is as follows. The data in your training and test/out-of-sample sets can be modeled as h(x) + noise. Here, the noise is the variability in your data that is not explained by some common (theoretically optimal) model h(x). The important thing here is that if your training and test data are sampled from entirely ...


0

Actually your last question, does regularization improve model performance, is a bit off. regularization is a way to reduce overfitting, not to check if you hit a plateau yet. As overfitting means your model try to fit even to noise in training data and not the real underlying pattern, the model will likely to perform bad on test data. That’s why ...


2

Here are some things that I think might help. If you are free to change the network architecture try using a large but shallower network. Layers help a network learn higher level features and by the last layer the features are abstract enough for the network to "make sense of them". By forcing training on a shallower network, you are essentially ...


2

Generally speaking, if you train for a very large number of epochs, and if your network has enough capacity, the network will overfit. So, to ensure overfitting: pick a network with a very high capacity, and then train for many many epochs. Don't use regularization (e.g., dropout, weight decay, etc.). Experiments have shown that if you train for long ...


1

Sampling is used during VAE training and during inference. The idea is that the data are encoded as random draws from a particular distribution. That distribution's parameters depend on the input data, because they are determined from the inputs. By contrast, an ordinary autoencoder maps the data to a vector deterministically, without any sampling. For a ...


4

Memorization For absolute overfitting, you want a network that is technically capable to memorize all the examples, but fundamentally not capable of generalization. I seem to recall a story about someone training a predictor of student performance that got great results in the first year but was an absolute failure in the next year, which turned out to be ...


4

There is no way to tell without knowing more about your situation. Yes, you may have hit a plateau. Or there may be hidden structure in your data that will only become apparent with (much) more data. Or some of your predictors may need to be predicted themselves, and you may be able to do a better job at that. Or you may not have included a hugely relevant ...


2

According to the Open AI paper Deep Double Descent, you need to have just a large enough neural network for a given dataset. Presumably this makes the NN powerful enough to perfectly learn the training data, but small enough that you don't get the generalisation effect of a large network. The paper is empirical, so the reason why it works is not theretically ...


1

Just reduce the training set to a few or even 1 example. It's a good, simple way to test your code for some obvious bugs. Otherwise, no, there's no magical architecture that always overfits. This is "by design." Machine learning algorithms that overfit easily aren't normally useful.


2

I like your question a lot. People often talk about overfitting, but may be not too many people realized that intentionally design an overfitting model is not a trivial task! Especially with large amount of data. In the past, the data size is often limited. For example, couple hundreds data points. Then it is easy to have some overfitted model. However, in &...


10

If you have a network with two layers of modifiable weights you can form arbitrary convex decision regions, where the lowest level neurons divide the input space into half-spaces and the second layer of neurons performs an "AND" operation to determine whether you are in the right sides of the half-spaces defining the convex region. In the diagram ...


9

I think the examples you bring are mostly from computer vision/image recognition and that case external datasets are very likely to include similar signal/dynamics as the prior data at hand. A "car" is a "car" irrespective of its surroundings. A "good customer" or "abnormal shopping activity" is different in Luxembourg ...


0

Quoting from TF's tutorial on RNNs: In addition to the built-in RNN layers, the RNN API also provides cell-level APIs. Unlike RNN layers, which processes whole batches of input sequences, the RNN cell only processes a single timestep.


9

At some point, adding more data will result in overfitting and worse out-of-sample prediction performance. Always. That papers report improved accuracy by leveraging additional data is not surprising at all. After all, people (both in academia and in industry) are heavily incentivized to report precisely this. Here is the relevant algorithm: 1. Pick an ...


0

Yes, this is the idea that the original paper promoted. Note, however, that it is a little bit tricky to use the term alignment. For purposes of the statistical machine translation, it was defined as a meaning correspondence of the source and target words. A highly cited 2017 study shows that the attention might learn very unintuitive alignments which seem ...


0

The Sound of AI YouTube channel has a series called Deep Learning for Audio in Python, and a newly started series Audio Signal Processing for Machine Learning. I keep some resources on Machine Hearing, including a list of Stack Overflow answers with code examples. The presentation Audio Classification using Machine Learning that I did at Europython was also ...


0

I'm also a newcomer and a self-learner in this field. I suppose there are two subfields: speech and "others", where "others" can include things like music. I have little experience with the "others" subfield so I will just focus on speech. I think the key thing to note is that audio, like images, can be converted to a series of ...


0

You could try https://www.coursera.org/learn/advanced-machine-learning-signal-processing#syllabus if you prefer videos or http://web.stanford.edu/class/ee269/slides.html if you prefer lecture slides.


0

The mask is simply to ensure that the encoder doesn't pay any attention to padding tokens. Here is the formula for the masked scaled dot product attention: $$ \mathrm{Attention}(Q, K, V, M) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}M\right)V $$ Softmax outputs a probability distribution. By setting the mask vector $M$ to a value close to negative ...


2

A bit of reference chasing, combined with Google Scholar searches, suggests that the origin was Japkowicz, N., Myers C., & Gluck M., (1995), “A Novelty Detection Approach to Classification”, in Mellish, C. (ed.) The International Joint Conference on Artificial Intelligence (IJCAI-95), Montreal, Canada. IJCAII & Morgan Kaufmann. San Mateo, CA. pp 518-...


0

Whenever you change the model configuration, you'll probably have to change the learning rate, any momentum parameters, and the number of epochs you use for training. I wouldn't be surprised if the best number of epochs, best momentum parameters and the best learning rate were different for each batch size. Additionally, if your objective function is a sum ...


26

In addition to the points raised in the other answers, a pruned network may not be faster. Common machine learning frameworks have very efficient optimizations for computing dense matrix multiplications (i.e. normal, unpruned layers), but those algorithms can't take any additional advantage of the fact that some weights are set to 0 (because they are pruned)....


9

As mentioned previously, you need to train on large networks in order to prune them. There are some theories as to why, but the one I'm most familiar with is the "golden ticket" theory. Presented in "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" by Jonathan Frankle, Michael Carbin the golden ticket theory of ...


38

Pruning is indeed remarkably effective and I think it is pretty commonly used on networks which are "deployed" for use after training. The catch about pruning is that you can only increase efficiency, speed, etc. after training is done. You still have to train with the full size network. Most computation time throughout the lifetime of a model's ...


0

In fact, the decision tree is equivalent with the deep neural network with the ReLU activation function. See the paper Oblique Decision Trees from Derivatives of ReLU Networks or the or discussion on decision tree and neural networks.


0

Indeed. You might want to read this paper comparing the 2 approaches on various NLP tasks: To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks The main conclusion is that results are quite similar in most cases.


2

With different network architecture, you may need different hyperparameters (e.g. learning rate, or batch size), the convergence speed will depend on those as well. Another hypothesis might be that the bigger network was more flexible because of having more parameters, so it was easier for it to adapt to the data. Smaller network offers a much more ...


3

"Validation loss" is the loss calculated on the validation set, when the data is split to train / validation / test sets using cross-validation. The idea is that you have three, separate sets of data: one used for training the model (train), one for doing things like hyperparameter tuning, model selection (validation), and one used to make final ...


0

I just wrote a blog post on that topic! In short, I write about different methods for dealing with the problem of sparse / irregular sequential data. Here is a short outline of methods to try: Lomb-Scargle Periodogram This is a way of computing spectrograms on non-equidistant timestep series. Data modeling with Interpolation networks You really don't want ...


4

This is a very deep question, because neural networks are very mysterious in this regard compared to classic learning algorithms. Modern applications of deep learning tend to use an enormous number of parameters, often much higher than the number of observations. As such, they will typically learn the training data exactly, and will achieve 0 error on the ...


Top 50 recent answers are included