New answers tagged

0

An example, with some number crunching, might be intuitive and also help debunk the idea of using the initial gradient instead of $0$. Consider the 1D problem $f(x)=x$, where $f'(x)=1$. $\beta_1=0.9$ and $\beta_2=0.999$ as usual. The first few values of $m_t$ and $v_t$ (rounded to 4 places) are given below. \begin{array}{c|c|c|c} t&m_t&v_t&m_t/\...


0

The problem is because of the existing overlap between the last record of train data and the first record of test data. You should simply drop the last time-stamp of train data! train = train.iloc[:-1, :]


1

Maybe you have too many layers. Try 2 or 3 layers instead of 15 and see what happens. I've noticed that if I give my neural nets too many layers, then they will fail to learn any relationship between the input and the output, and will instead simply output a constant value. The easiest solution is to use fewer layers. The problem could also be solved by ...


0

I see two statistical problems here that can be worked around with a different approach. The first is using a split sample for model building and evaluation. The risk with the split is that you lose both power in building the model and precision in testing the model. Frank Harrell estimates that you need at least 20,000 subjects to get around those combined ...


0

Convolutional and fully connected layers are the building blocks of most neural networks. They are the units (layers) that most NNs are constructed from. Convolutional and fully connected layers are multiplication parameters that connect one layer of neural network to subsequent layers, thereby making each layer’s weights as a linear combination of its ...


1

A practical approach to take while training your model might be: Set a maximum number of iterations before the episode ends, some value that's probably more than sufficient to achieve an optimal solution. Let the agent terminate the episode at any point in time. Add a small negative reward at every point in time. This should encourage the agent to ...


0

Here is a more information-theoretic reason for why this is happening. Let $Y$ be the true and correct output that your network should return (the target), and let $\hat{Y}$ be the output that your network actually returns. The problem that you observe is that $\hat{Y}=K$, where $K$ is some constant, which means that $Y$ is independent of $\hat{Y}$. If we ...


2

Let $r(x)$ denote RELU function. For $x\geq 0$, $r(x)=x$, so nothing changes on the RHS plane. The number of linear pieces stays the same on RHS. For $x<0$, $r(x)=0$, so the number of linear pieces is $1$. That means, while we'll have the same number of linear pieces for the RHS, we'll decrease the number of linear pieces to $1$ for the LHS. So, in ...


0

There is no direct relationship between alpha and accuracy. The alpha influences the weights of the model. And the weights influence the loss of the model. Finally the loss influences the accuracy of the model. The change of alpha will certainly lead to the change of the value of the weight and the loss, but different weights and loss don't mean different ...


0

Check your software, but broadly the answer to your question is: yes, using batch normalization obviates the need for a bias in the preceding linear layer. Your question does a good job of laying out where you are confused, so let me speak to it: the shift term in batch normalization is also a vector, for instance the documentation for BatchNorm2d in Pytorch ...


0

There are situations where (input_dim + 2*padding_side - filter) % stride == 0 has no solutions for padding_side. The formula (filter - 1) // 2 is good enough for the formula where the output shape is (input_dim + 2*padding_side - filter) // stride + 1. The output image will not retain all the information from the padded image but it's ok since we truncate ...


1

It depends on how advanced you are. I would recommend William Bolstad's "Introduction to Bayesian Statistics" and his "Understanding Computational Bayesian Statistics." In Pearson and Neyman's method of statistics with its minimum variance unbiased estimator, uniformly most powerful tests, and so forth, all information comes from the ...


0

Machine Learning: A Bayesian and Optimization Perspective (2nd Edition) by Sergios Theodoridis seems to squarely address your interests: it builds from topics you know about probability theory to topics in machine learning it covers a wide range of machine learning topics it motivates machine learning topics from the perspective of Bayesian reasoning


2

The problem is that the gradient is not small for all layers, but small for the deeper layers (further away from the output). For the layers near the output, the gradient may be big. If you now have a huge learning rate, the weights in the near-to-output layers will shoot around erratically and the net will not converge.


0

If Global Meanpooling is: $$h = \frac{\sum^p x_i}{p}$$ It's adjoint is $$\frac{\partial h}{\partial x_i} = \frac{1}{p}$$ A constant array. So, an operation based on that adjoint inverts, in a sense, the Global Meanpooling. If you have a scalar $y$, the resulting activation $\mathbf g$ (a vector in the 1D case) is: $$\mathbf g = \frac{\mathbf 1}{p}y$$ This is ...


0

Queries is a set of vectors you want to calculate attention for. Keys is a set of vectors you want to calculate attention against. As a result of dot product multiplication you'll get set of weights a (also vectors) showing how attended each query against Keys. Then you multiply it by Values to get resulting set of vectors. Now let's look at word processing ...


1

That's what going on in the filter of a convolutional neural net. It forces weights to be equal, hence the common colors going to each neuron in the hidden layer on the right, and it drops (sets to zero) some of the weights. In contrast, a fully-connected layer would have a unique color per pixel-neuron pairing, meaning $9\times 4=36$ colors, rather than the ...


0

You may want to have a look at "Unconstrained Monotonic Neural Networks". The basic idea is to construct a neural network that forces the output to be positive. The integral of the output from that neural network is the final output that forms a monotonic function. The paper describes how to train such a neural network. That is to say how to get ...


0

Short answer: Yes, you can and should always report (test) MAE and (test) MSE (or better: RMSE for easier interpretation of the units) regardless of the loss function you used for training (fitting) the model. Long answer: The MAE and MSE/RMSE are measured (on test data) after the model was fitted and they simply tell how far on average the predictions were ...


0

You’ve correctly understood $w_j$ and $w_I$: they’re the target word and input word. Section 1.2 goes into more detail about the CBOW case; I’m filling in the pieces that weren’t explicit. The LHS of the equation would be different: $$p(w_j \mid w_{I1}, \ldots, w_{IC})$$ to show that you’re conditioning on several input words. The RHS is also different, ...


0

I would say that you do not necessarily have an over-fitting problem - it rather depends on what you take over-fitting to mean. Some would define over-fitting as meaning the performance on the training data is substantially better than the performance on the validation/test data. However this is not a particularly useful definition as many machine learning ...


0

i found the answer to my question in this video https://www.youtube.com/watch?v=p3CcfIjycBA&ab_channel=DigitalSreeni Very clear and simple.


0

You might want to read this paper: https://arxiv.org/abs/1905.10617 "The exposure bias problem refers to the incrementally distorted generation induced by the training-generation discrepancy, in teacher-forcing training for auto-regressive neural network language models"


1

Seems that some people are a bit confused by your question because of its title; indeed, RBF neural networks exist, but they are a different architecture than the traditional multi-layer NNs. The body of your question is complete and formulated well (apart from some ambiguity because of the use of MLP), so to get to the answer let me ask a slightly different ...


5

This answer depends on the definition of artificial neural network (ANN) you take to be true. See my question here: What *is* an Artificial Neural Network?. Therefore, no objective answer can be given since: To accommodate all definitions of ANNs, they are simply defined as arbitrary computational graphs, with tunable parameters (even if they are not tuned, ...


0

Deep learning is a subset of machine learning, which is a field dedicated to the study and development of machines that can learn, and the goal is of deep learning is to achieve eventually attain general artificial intelligence. Neural network is just one of the biological inspired model. In simple terms I would define deep learning a subset method in ...


16

I'm going to disagree with the other answers. Fundamentally, I would say that deep learning is defined by a hierarchy of learned representations, and not by which particular model is used to define these representations. Indeed, this is how Goodfellow et al define it in the introductory section of their text Deep Learning (neural networks are not mentioned ...


4

Deep learning is machine learning done using “deep” neural networks, i.e. such that have multiple (>2) layers. So you cannot do it without neural networks. For using other kinds of machine learning just use “machine learning” term, that includes neural networks as well.


1

Yes. The input\output of VAE doesn't need to be gaussian. The decoder can take the multivariate gaussian and transform it to other (possibly) non-gaussian distribution. The latent space of VAE is gaussian because of the following properties: It is continues, which is suitable for generating objects which are continues by nature, and enables continues ...


28

This is a good question. Is a neural network essential for deep learning? Yes, your teacher provided you with a correct definition of deep learning. You can still do machine learning (a broader category) without neural networks, but you need a neural network for it to qualify as 'deep learning'. Isn't it possible to do deep learning without a neural ...


2

Why not directly minimize the Jensen-Shannon divergence between the generator and empirical distribution? Because it's intractable to compute. The marginal distribution $p(x) = \int p(x|z)p(z) dz$ is very hard to work with, computationally.


0

It is quite easy to mess up the first stage model or to fail to see the leakage of informations when working with large blends of models. As stated by @Erin LeDell you should make sure that the second stage is learned from cross validated predictions of the first stage. I wrote the following tutorials regarding blending if you are interested: Introduction to ...


1

Tensor = multi-dimensional array In the machine learning literature, a tensor is simply a synonym for multi-dimensional array: Tensors, also known as multidimensional arrays, are generalizations of matrices to higher orders and are useful data representation architectures. Tensors in Statistics Annual Review of Statistics and Its Application (2021) Hence ...


1

Neural networks require gradients to be trained. If you would do classification as you described, i.e., $$\hat{y} = f(s) = \begin{cases} 0 & s \leq 1 \\ 1 & s > 1 \end{cases},$$ it would be hard to define a gradient w.r.t. $s$, which is the output of the network. Therefore, you do not want to use classification in this sense. Instead of pure ...


0

I agree the authors didn't do a great job of clarifying this. My best guess is that the regression model looks like: $y_i \sim \mathcal{N}(\mu=f(x_i), \sigma^2 = g(x_i))$ where $f$ and $g$ are neural networks (or more precisely, probably two branches of the same neural network). I'm not sure I understand your concern about the uncertainty bands on the BBB ...


2

What do these weights learn? At the most literal level, they learn to predict the tokens of a sequence of text, given a corrupted form of that same sequence. In other words, they learn common patterns in language. BERT is a gigantic pseudo–language model. Are weights learned for each token? It’s easy for a misunderstanding to arise here. For the same word ...


2

The word embeddings are weights in the model. They get learned just like any other. What you input to your model isn't a word embedding. Instead, you provide one-hot vectors for each word. These get multiplied by the matrix of embeddings to select the embedding for each word. That matrix is a parameter of your model. Equivalently and more space-efficiently, ...


0

One possible and simple approach in your case is the one-shot recognition. The idea is to use the pre-trained library that extracts the feature vector from the face images. So, you compile a DB with the unique vectors for each of your client faces. These vectors are in Euclidian space, meaning that you can simply calculate the distance between the images, ...


1

If input data is not normalized (or standardized), it will not work in a Neural Network. This is actually a numerical problem, due to the gradient update but in general one has to normalize or standardize data when dealing with deep learning. So probably that it will work if you normalize your data and make the last layer a linear one rather than sigmoid. ...


0

In the neural network context, the phenomenon of saturation refers to the state in which a neuron predominantly outputs values close to the asymptotic ends of the bounded activation function. Measuring Saturation in Neural Networks (2015) So, saturation refers to behaviour of a neuron in a neural network after a given period of training/for a given range ...


1

This is completely possible. For instance, you can look into Siamese network. However you will have to most likely recode a custom architecture and a custom loss fonction, if your problem is not a "classic" one.


1

Are the attention weights based solely on position of the tokens? No, they're based on the specific values that appear in the sequence. The attention formula actually has no information about position at all. Because in natural language processing we believe the order of words is important, we will often incorporate a positional encoding into the vectors we ...


2

They got the first equation by taking the log of the normal Bayes rule expression, $$ p(\theta \mid D) = \frac{p(D \mid \theta) p(\theta)}{p(D)} \text{.} $$ To make it clear that this is a general claim (not specific to $D$ and $\theta$), I could rewrite this as $$ p(B\mid A) = \frac{p(A \mid B) p(B)}{p(A)} \text{.} $$ If this is unfamiliar, rearrange these ...


2

There are two parts here: why MLPs are no good and what NLPers like about RNNs. A multi-layer perceptron can only process fixed-size inputs. In your example up above, you could indeed pass [3, 1, 20] (‘cat’) to an MLP if it has 3 input nodes. But you can’t give the MLP [3, 1, 20, 19] (‘cats’)! You’d need to add another input node (and associated weights), ...


1

Some thoughts on your interesting question: Good features are problem dependent. So seems difficult (if possible) to incorporate feature engineering into any rigorous mathematical framework. To me, pushing the problem of finding good features to finding good kernels is a way to separate the problem in two parts. First, the mathematics (learning from good/...


3

Is $D(x)$ here considered as a probability distribution? Yes. In fact, it is the probability of label or class given an input feature vector or image, denoted as $p(c|\mathbf{x})$. Here are a few resources to better understand this: the relationship between maximizing the likelihood and minimizing the cross-entropy Machine Learning: Negative Log Likelihood ...


1

You ask in a comment: Let's say we backprop the matrix. If we do that we will get 32 gradient updates for each parameter in the neural net. For each parameter, can't we just take the average of the 32 gradients and use that in gradient descent? Well, suppose you have 1,000,000 parameters. You're suggesting that we calculate 32,000,000 partial derivatives ...


0

Due to the underspecification problem, it's hard for us to calculate the sample size because we don't know if the model performs well by skill or just by luck. But we can start off with the sample size calculated using the method I provided in the previous answer and do bootstrap sampling several times each time with a larger sample size. We can check the ...


0

A model is overfitting if the model conforms too closely to your particular choice of training data. In other words, suppose you train a model M. M is overfitting if your metric for model evaluation (in this case accuracy) does much better on your train set than your test set. Now, some reduction in model performance moving from your train set to your test ...


3

Latent space is a vector space spanned by the latent variables. Latent variables are variables which are not directly observable, but which are $-$ up to the level of noise $-$ sufficient to describe the data. I.e. the observable variables can be derived (computed) from the latent ones. Let me use this image, adapted from GeeksforGeeks, to visualise the idea:...


Top 50 recent answers are included