New answers tagged

0

Note that the weights in word embeddings get initialized with zero mean and unit variance. Also, the embeddings get multiplied by sqrt(d) before being added to the positional encoding. The positional encoding is also on the same scale as the embeddings. My hypothesis is that the authors tried rescaling the embeddings by various numbers (as they certainly did ...


0

The linear layer processes each element in the sequence independently, there is no interaction between them. Therefore information about future tokens is not available, also during training. Interaction/information exchange between tokens only occurs in the attention layers (hence we need masking)* (* I have not really looked into the normalization layers ...


0

The sublayers refer to the self/cross multi-head attention layers, as well as the position-wise feedfoward networks. Your code is mostly correct, but: your pseudocode accidentally overwrites the value of the original x. The layer norm is applied after the residual addition. there's no ReLU in the transformer (other than within the position-wise feed-forward ...


0

Neural networks can do lots of things and you can probably get one to somehow do this, but this does not really sound like a problem where a neural network would be the obvious choice. It would seem more obvious to describe this kind of data using a mixture distribution. I.e. you have some kind of distribution such as a lognormal, normal or uniform (or ...


6

The Bitter Lesson is that in the long term, progress is dependent on leveraging more and more computational power. This is not to say that algorithmic and modeling progress isn't important, but they aren't the limiting factor -- neural networks have been since the 1950s (or earlier), and it's only now that increasing computation resources have let us ...


3

The location address mechanism doesn't focus on a single address, but outputs a distribution on addresses to focus on. This is necessary because otherwise the process would be nondifferentiable, and gradients wouldn't be able to flow backward through the addressing mechanism, and you wouldn't be able to train it. However, if you repeatedly convolve these ...


2

It's impossible to "losslessly" visualize a very high dimensional space in just 2 dimensions. One approach is to randomly pick two unit vectors $u, v$ in $\mathbb{R}^n$, and then you can "project" your loss $f(\vec x)$ into the easily visualized, two dimension $g(a,b) \mapsto f(au+bv)$. Hao Li et. al present a slightly more refined ...


1

In most applications, CNNs treat pixel values as a continuous variable ranging (without loss of generality) from 0 to 1. There's no substantial difference between mapping 256 discrete values onto a continuous range, versus mapping 65536 values onto this continuous range. NB - "high dynamic range" is an very overloaded term. Literally, it means &...


2

Curriculum learning (possibly coined here) is a commonly used term for gradually increasing the difficulty in some learning problem. It's often used in reinforcement learning applications.


1

In order to train a neural network with backpropagation / gradient descent, it's generally necessary for the loss to be differentiable with respect to the parameters of the neural network. If you have a differentiable physics simulator, you can have a loss which depends on the output of the simulation. However, if your physics engine is not differentiable ...


0

use the function sobolshap_knn in R package "sensitivity" analysing global sensitivity with categorical and discrete variables. First, you create dummies, for example, if you have three colours, white, red and blue, then then create three dummies X1,X2,X3, then use (X1=1,X2=0,X3=0) to represent white, (0,1,0) represents red, (0,0,1) represents blue....


13

Disclosure: I didn't have time to carefully read the full paper yet. From the abstract: Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. Foundational models is a term with a good hype potential, but ...


0

This approach often used in NLP models, so called via tuning and it is what you called transfer learning. A recent debate on this so called Foundation models actually revolves around this concept. transfer learning is even really used on autoencoders? Yes, it is possible, as these new foundation models approach training a large vision models or ...


0

I had the same problem, restudy the equations, and found that in my coding, I mixed up the n, size of the whole training set, with the m, size of the mini batch. Based on my understanding, for the weight decay term calculation, n, size of the whole training set should be used. Whereas, m, size of the mini batch is used in the approximation of gradient of ...


-1

You need to define how you intend to "compare" two neural networks. The most obvious route would be to compare how they encode your input variables into a latent space z, gather samples, estimate a distribution, bada bing. You could also compare them on accuracy. Get my point?


2

Indeed relu is also bounded below, they didn't claim otherwise. The difference is, that swish allows small negative values for small negative inputs, which according to them, increases expressivity and improve gradient flow. The reason behind improving generalization is that, as in regularization, small, approaching zero, weights improve generalization as ...


2

Yes. Before mini-batching existed, SGD referred specifically to batch size equal to one. You can actually use a bigger batch size though, you just need to add gradients from sequential samples within a batch. This is called Gradient Accumulation. See link.


1

This is answered on page 2 of "Explaining and Harnessing Adversarial Examples" by Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy. The purpose of the attack is to find examples from class $i$ that are misclassified as class $j\neq i$ by adding some noise $\eta$ to the input, where $\| \eta \|_\infty $ is "small" in a specific ...


0

There a whole bunch of ways you could go about this. Overfitting is definitely a danger if you have too many features. A few suggestions: Rather than using all of the other stocks in the sector and potentially exploding the number of features in your dataset, try constructing some indicator features (e.g. average historic returns in sector, average ...


1

As Sycorax suggested, you have to use a softmax activation function since you have 4 classes and your loss function expects probabilities. Two ways to do this: Specify the activation of your dense layer output = layers.Dense(model_classes, activation='softmax')(x) Add an additional layer after the Dense layer: final_layer = Activation('softmax')(x). This ...


0

Let’s say the set of functions that a neural network with k layers can learn is F (this is called your parameterized function space, since it’s all the functions with the parameters of the weights of your network). With k+1 layers, the set of functions you can learn, let’s call it F*, is all the functions in F, plus any new functions from the new layer (just ...


1

output = layers.Dense(model_classes)(x) measures the predictions on the wrong scale. categorical_crossentropy expects a probability, but output is not restricted to probabilities. Add a softmax activation to the final layer; this will fix the problem because softmax yields probability vectors.


3

Michael Betancourt writes a little about this in this paper. Here is an excerpt of his paper: This assumption implies that the variation in the integrand is dominated by the target density, and hence we should consider the neighborhood around the mode where the density is maximized. This intuition is consistent with the many statistical methods that ...


2

The difference is that with full posterior you estimate the distribution of the parameters. With MAP you find only the mode of the distribution. You cannot really compare them, it’s like asking how all the people’s age differs from the average age of the whole population. It obviously does. Another difference is practical, you use different algorithms for ...


1

I think the Keras documentation gives a good explanation stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. In other words, when stateful=True, Keras saves the state between batches so that you don't have to. This blog post has more ...


0

The formula you've given is for 1 filter. If you have $N$ filters, then you have $N$ outputs of the size you describe.


1

They are equivalent up to computational issues that arise from doing math on a computer. $$R^2=1-\dfrac{SSResiduals}{SSTotal}$$ $$ RMSE = \sqrt{\dfrac{SSResiduals}{n}} $$ Both are just functions of the sum of the squares residuals (“errors” in a mostly-acceptable-even-if-technically-incorrect machine learning slang). Setting aside some issues that can arise ...


0

If you want $(\Phi^T\Phi)^{-1}$ to be invertible (where $\Phi \in R^{N \times K})$ you need to ensure $rk(\Phi) = K$ (i.e. full rank). As mentioned in the comments, if $\Phi$ is rank deficient then $\Phi^T\Phi$ is positive semidefinite and thus not always invertible.


1

The validation data should not be influencing the training process. The validation data is used to a "sanity check" of the model, and for us to check for overfitting or undefitting. So for example, if you have a overfitted model, the training accuracy will be really high, because the model "memorized" the data, and the validation accuracy ...


0

Yes. This happens even when people don't want it to. It happens when people train catagorization networks with very unbalanced datasets, then network will often catagorize every input as the most common class in the training set. It also happens in GANs, with so called "mode collapse", where the generative portion of the network learns to produce a ...


0

I'd probably go for the Brier score (the mean squared error on the labels) rather than the cross-entropy loss as the cross-entropy loss is likely to be dominated by very confident miss-classifications, which are likely to be far from the decision boundary (so some proper scoring rules are better than others - you want one that gives relatively more attention ...


3

Other answers say that traditional regression models, like linear or logistic regression, already does this, as regression is to model conditional expectations (or conditional probability, conditional hazard, conditional ...). As soon as you are calculating a prediction interval with some regression model, you are entertaining some kind of probabilistic ...


2

It'll have more weights and in each new added layer you're getting more complex functions that's in NN sense your function will be a more complex composition of another functions and your output will be hardier to optimise so its convergence will be slower. I'd say that convergence getting slower is mainly due to complexity composition, the number of weights ...


0

In the context of machine learning, is there any difference between the terms unit and neuron? They are the same, often called neural unit. Neurons in an ANN is derived from the McCulloch-Pitts Neurons(MCP neuron), and a MCP neuron is a highly simplified model of a neuron in the human brain. In 1943 Warren S. McCulloch, a neuroscientist, and Walter Pitts, ...


2

The notation convX_Y refers to the identification of the convolutional layer. See the VGG-16 network as an example. The convolutonal layers that occur in sequence prior to a pooling layer have the same X identifier and different Y identifiers. After the pooling layer, the X identifier is incremented, and this naming convention is repeated. https://icmlviz....


3

Yes, we can. there are many ways to initialize the weights for neural network. In fact, I do not agree with "bias neuron is always initialized to 1". Search neural network weight initialization in google scholar to get more information.


1

The different words in a sentence can relate to each other in many different ways simultaneously. For example, distinct syntactic, semantic, and discourse relationships can hold between verbs and their arguments in a sentence. It would be difficult for a single transformer block to learn to capture all of the different kinds of parallel relations among its ...


3

Short version: the character representations are there so you can still embed tokens that were never seen during training. Recall that embedding an (atomic) object is done by selecting the corresponding vector in a lookup table. ELMo has a token embedding for "cat", for "the", for "likewise", and so on. All of those exist in a ...


2

Weights determine slopes of the activation functions. Regularization reduces the weights and hence the slopes of the activation functions. This reduces the model variance and the overfitting effect. The biases have no influence on the slopes of activation functions. However, they have an influence on the position of the activation functions in space. Their ...


0

I think concatenating the two arrays won't work with either architecture as the time dimension would not be separated from the data point itself, making it difficult for the network to learn from these two different aspects. Let's say a single training example from your description has a shape (1, 10). If I understood correctly, if you want to add another ...


1

Cross entropy loss by Softmax is a loss function. If you try [0, 1, 0] and do (y-pred) you would use another loss function. Why not just 0 and 1? Sometimes, the loss function we actually care about (say, classification error) is not one that can be optimized efficiently. For example, exactly minimizing expected 0-1 loss is typically intractable (exponential in ...


1

I would compute dense feature vectors for the two strings separately, then concatenate, then use a FCN to make the classification decision. In psuedocode, it would look something like representation1 = additional_layers(embed(string1)) representation2 = additional_layers(embed(string2)) combined = concatenate([representation1, representation2]) y_hat = ...


0

Instead of having padded convolutions to maintain the spatial size of the feature maps, they pad the original image by mirroring the borders and forward the pre-padded image through the network I agree with @bogovicj 's explanation. What do they mean by "only us[ing] the valid part of each convolution"? Based on your understanding of @bogovicj '...


2

The three signals definitely need to be treated separately. Also, you haven't specified how you intend to model this, but I'd suggest some dimensionality reduction technique. As I see in the figure, the right muscles have very abrupt jumps, while L-TA has more natural oscillations. This would likely be easy to see in the Fourier transform, where the abrupt ...


8

Whether you need a softmax layer to train a neural network in PyTorch will depend on what loss function you use. If you use the torch.nn.CrossEntropyLoss, then the softmax is computed as part of the loss. From the link: The loss can be described as: $$ \text{loss}(x,class) = −\log\left(\frac{\exp⁡(x[class])}{\sum_j \exp(x[j])}\right) $$ This loss is just ...


3

Neural Networks with two layers and enough neurons can theoretically approximiate every function. The reason of doing it deeper - meaning more layers - is that it became clear that those networks learn better and with less data. Those networks start to build more abstract filters with each added layer. The whole point of doing backpropgation and deep ...


3

You have several mistakes. You're missing the minus sign due to $-\hat y_i$ term If you have $n$ samples, $x_1$ and $x_2$ should also have indices Derivative wrt $b_3$ can't be $0$ because it's incorporated in the loss term. From what I see, $\frac{\partial \hat y}{\partial b_3}=1$ Your theta vector is incomplete, I believe it should be something like below:...


2

Batch gradient descent (GD) (or any GD) can run into local minima problem because the algorithm is not exact, it is a heuristic. The paragraph starts with assuming that $S$ is a collection of the same samples in $S_{sub}$ repeated 10 times. So, the minima don't change. It also states that, in large-scale real data, we don't have exact duplicates, but there ...


0

I believe you mean output activation by in front of the MSE. In logistic regression, cross entropy is used for the loss function, not MSE (mean squared error). But, independent from the loss function, the gradient portion produced by the sigmoid will contain $\sigma (1-\sigma)$ multiplier, and if $\sigma$ was $1$, the gradient would be $0$ irrespective of ...


1

They're some sort of generative models, learning the PDF so that they can sample from it. This is achieved by having random latent representations and mapping them to the relevant domain. So, you can easily generate new samples, but, this is not the same thing as calculating the probability of that sample. The learnt parameters belong to the neural network, ...


Top 50 recent answers are included