# Tag Info

1 vote

### I need to get 100% accuracy on my training data

What you are describing is that you need to losslessly compress and retrieve the data. Did you consider any off-the-shelf caching solution? Since you care about “predicting” the seen data, it's about ...
• 127k
1 vote

### Why do the error derivatives become small if we start with a large learning rate?

So in the case of sigmoid neurons, having large weights means the hidden unit output saturates, so then changes in the weights have minimal effect on the hidden unit output, and therefore the error ...
• 5,354
1 vote
Accepted

### Intuitive difference between NN and attention for text prediction

Neural networks are universal approximators, so they could approximate any function. But for it to really work you possibly would need a huge number of weights, a lot of data, and training it for a ...
• 127k

### How to calculate influence of variables at ROW LEVEL?

What you're trying to do is far from a solved problem. My impression is that the scientific consensus on model-agnostic interpretability ranges from promising to problematic to fundamentally unsound. ...

### Why does a neural network have the same output for every item in a batch?

I met the same issue as you. I tried to fine-tune a large language model with more than millions of parameters but it outputs exactly the same for each batch. Finally, I figured out that I used a too-...
• 21
1 vote

### Comparison of the performances of Regression Models and ANN models

Out-of-sample testing is the standard way to do this. Train your model on most but not all of your data Even better might be to have multiple out-of-sample groups (something like cross validation). ...
• 46.4k

### What is R squared for a neural network and what does it signify?

Especially in complicated setting, the exact definition of $R^2$ is not clear. The definition in the question is close to the definition I like and that sklearn ...
• 46.4k
1 vote

### What does the error in artificial neural network stand for, is the same with mean square error (MSE)

It depends on what loss or error function you use when you code the network. If your loss is the sum of squared errors (SSE), then divide this value by the number of predictions being made. If your ...
• 46.4k

### Are a model's predictions (such as for stock prices) always correct at a global minimum?

Let's do an example. ...
• 46.4k

### Is more data really always better in machine learning?

You are right, it is not only about the size of the dataset. As two other answers pointed out, having more data (vs very little) is desired, as even in a noiseless scenario it may help you to get a ...
• 127k

### Is more data really always better in machine learning?

My intuition is that, given $(x_{i},y_{i})_{i=1}^n$ and $(x_{i},y_{i})_{i=1}^N$ have the same "information" (I know this is a fuzzy term), using $(x_{i},y_{i})_{i=1}^N$ should not better the ...

### Neural Networks - Can I Use Any Activation for the Output Layer?

universal approximation theorem says that using linear output (and other assumptions) you can approximate any function, however if you have a bounded output, you can for sure use bounded activation... ...

### Is more data really always better in machine learning?

Intuitively, having more data will tell the neural network where to turn, by how much, and in what direction (up/down, left/right, combinations, extensions in high-dimension spaces, etc). Imagine your ...
• 46.4k
Accepted

### Does stacking multiple linear layer have some documented improvements?

Here are some papers giving theoretical guarantees for Multiple Linear Layer Networks (more generally known in the literature as Deep Linear Networks) which should be of interest to you : First, the ...
• 992

### Which would be a better approach for Multiclass segmentation?

The best approach would typically be an ensemble (2) of multi-class segmentation models (3). However, the best next step would be to just see how well a multi-class model (i.e. using softmax) works ...
• 1,786

### How to express backpropagation dE/dV using matrix

I can directly enter the wjk matrix and zj matrix, but for xi, I only have to enter x1 while the xi matrix consists of x1, x2, and x3. $W_{jk}$ is not a matrix, it's a real number whose value ...
• 54.5k
1 vote

### Viewing a neural net loss function as a Gaussian Process

Neural Networks as Gaussian Processes Consider a neural network with only one layer (i.e. no hidden layers, i.e. logistic regression): \operatorname{reg}: \mathbb{R}^N \to \mathbb{R}^M : \boldsymbol{...
• 1,786

### Viewing a neural net loss function as a Gaussian Process

Disclaimer: I just glimpsed at the linked articles. My answer focuses solely on Gaussian processes per se. Your example If you understand your example $f(x)=x^2+Y$ as a random function of $x$, then it ...
• 2,247
1 vote
Accepted

### How do LLMs transform tokens into vectors?

Yes, it is exactly as you say. LMs (and machine translation models, too) start with a randomly initialized embedding matrix, which is learned via standard backpropagation. It is typically not ...
• 2,946
1 vote

### Using 1-Layer Fully-Connected Neural Network to Appoximate Exponential Functions

I think your professor means that $e^x$ cannot be approximated by a single layer of ReLU globally (on the entire $\mathbb{R}$), which seems correct because whatever the output of a single layer of ...
• 181

### Having more feed forward layers as hidden layers in word2vec

Word2vec is popular because it's simple while being good enough. Because of being simple, it's fast and needs less expensive hardware to run. Yes, using one vs two layers does not seem to make much ...
• 127k

### Covering number of a single-layer neural network with Lipschitz Function

See this paper: https://arxiv.org/abs/1706.08498 for a solution, which introduces a covering bound on matrix-matrix products together with an inductive technique to combine covers of layers.
Accepted

### Very balanced dataset and a multiclass classification problem, no context behind the inputs. Which evaluation metric to use?

The trouble with accuracy is that your model does not predict discrete classes. The neural network outputs values on a continuum that have a (granted, weak) interpretation as the probabilities of ...
• 46.4k
Accepted

### question about mlp() in tidymodels

It's a model with a single hidden layer, plus the input and output layers, with all nodes connected from one layer to the next. That's the simplest case of multilayer perceptron (see eg Wikipedia). ...
• 28.9k

### question about mlp() in tidymodels

I agree that the explanation is confusing. The a.k.a. section refers to "perceptron model", not "multilayer perceptron model". You can sense this from the tinymodel definition of ...
• 54.5k

### Bayesian Linear Regression on the top of deterministic neural network

There's a simple option, if you are fine with maximum-a-posteriori estimation and prediction, but don't care so much about uncertainty. I guess you don't, because it's hard to see how you'd get that ...
• 27.1k

### How to determine the no of multiplication operations in convolution operation?

Sorry, but gunes made some mistakes that I have to point out. For each output pixel, we actually have 5 x 5 x 192 multiplications. The output size is 28 x 28 x 32 (not 28 x 28 x 192). The thing you ...
Accepted

### Why must a product of symmetric random variables be symmetric?

To say that a random variable $W$ "has a symmetric distribution around zero" is saying that $W$ and $-W$ have the same distribution. Let $X$ be another random variable and set $Y=WX.$ By ...
• 306k
Accepted

### which one between XGBoost and neural networks is more interpretable?

Both (sufficiently large) neural networks and XGBoost are not interpretable on their own (they are not "intrinsically interpretable") and are typically seen as part of the same "not ...
• 1,846
1 vote

### Clarification on the connection between deep ensembles and bayesian neural networks

Bayesian neural network is a fake technology. Take any example, which shows distribution of the target and change data into multimodal, just split them into two blocks with the gap between, launch BNN ...
1 vote

### What are alternatives of Gradient Descent?

(1) Bipropagation is a semi-gradient descent algorithm much faster than backpropagation. It solves the XOR problem each time and is 20 times faster than the fastest attempt of backpropagation. (2) ...
Accepted

### How to initialize final layer to get a good starting loss?

It can be done using the bias of the final layer of the network. Here I'll show how to derive it for a balanced classification problem, but the same can be done for unbalanced problems or regression ...
• 274

### What is the origin of the autoencoder neural networks?

The first clear autoencoder presentation featuring a feedforward, multilayer neural network with a bottleneck layer was presented by Kramer in 1991 (full text at https://people.engr.tamu.edu/rgutier/...

### How to initialize final layer to get a good starting loss?

You could use the Xavier initialization described in this paper. It allows the output of the classification layer (fully connected + softmax activation function) to be close to a uniform distribution, ...

### How to calculate the decay rate given an initial learning rate and final learning rate for schedulers when training neural networks?

I ended up figuring it out. For the exponential decay, it was easier than I thought, as the formula for the decay is ...
1 vote
Accepted

### Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)

It is actually the same cross-entropy as with current language models; only the notation is different. It is well illustrated in Figure 1 of the paper: Function $f$ already returns the $i$-th index ...
• 2,946
1 vote
Accepted

### How does masking work to make neural networks use varying input lengths

Masking is not enforcing weights to be zero, masking enforces network inputs and intermediate hidden states to be zero. Setting the inputs to be zero at the begging is not enough; many places in a NN ...
• 2,946

Top 50 recent answers are included