New answers tagged neural-networks
1
vote
I need to get 100% accuracy on my training data
What you are describing is that you need to losslessly compress and retrieve the data. Did you consider any off-the-shelf caching solution? Since you care about “predicting” the seen data, it's about ...
Tim♦
- 127k
1
vote
Why do the error derivatives become small if we start with a large learning rate?
So in the case of sigmoid neurons, having large weights means the hidden unit output saturates, so then changes in the weights have minimal effect on the hidden unit output, and therefore the error ...
- 5,354
1
vote
Accepted
Intuitive difference between NN and attention for text prediction
Neural networks are universal approximators, so they could approximate any function. But for it to really work you possibly would need a huge number of weights, a lot of data, and training it for a ...
Tim♦
- 127k
0
votes
How to calculate influence of variables at ROW LEVEL?
What you're trying to do is far from a solved problem. My impression is that the scientific consensus on model-agnostic interpretability ranges from promising to problematic to fundamentally unsound. ...
- 1
0
votes
Why does a neural network have the same output for every item in a batch?
I met the same issue as you. I tried to fine-tune a large language model with more than millions of parameters but it outputs exactly the same for each batch. Finally, I figured out that I used a too-...
- 21
1
vote
Comparison of the performances of Regression Models and ANN models
Out-of-sample testing is the standard way to do this. Train your model on most but not all of your data
Even better might be to have multiple out-of-sample groups (something like cross validation). ...
- 46.4k
0
votes
What is R squared for a neural network and what does it signify?
Especially in complicated setting, the exact definition of $R^2$ is not clear. The definition in the question is close to the definition I like and that sklearn ...
- 46.4k
1
vote
What does the error in artificial neural network stand for, is the same with mean square error (MSE)
It depends on what loss or error function you use when you code the network.
If your loss is the sum of squared errors (SSE), then divide this value by the number of predictions being made.
If your ...
- 46.4k
0
votes
Are a model's predictions (such as for stock prices) always correct at a global minimum?
Let's do an example.
...
- 46.4k
5
votes
Is more data really always better in machine learning?
You are right, it is not only about the size of the dataset. As two other answers pointed out, having more data (vs very little) is desired, as even in a noiseless scenario it may help you to get a ...
Tim♦
- 127k
5
votes
Is more data really always better in machine learning?
My intuition is that, given $(x_{i},y_{i})_{i=1}^n$ and $(x_{i},y_{i})_{i=1}^N$ have the same "information" (I know this is a fuzzy term), using $(x_{i},y_{i})_{i=1}^N$ should not better the ...
- 61k
0
votes
Neural Networks - Can I Use Any Activation for the Output Layer?
universal approximation theorem says that using linear output (and other assumptions) you can approximate any function, however if you have a bounded output, you can for sure use bounded activation...
...
5
votes
Is more data really always better in machine learning?
Intuitively, having more data will tell the neural network where to turn, by how much, and in what direction (up/down, left/right, combinations, extensions in high-dimension spaces, etc).
Imagine your ...
- 46.4k
3
votes
Accepted
Does stacking multiple linear layer have some documented improvements?
Here are some papers giving theoretical guarantees for Multiple Linear Layer Networks (more generally known in the literature as Deep Linear Networks) which should be of interest to you :
First, the ...
- 992
0
votes
Which would be a better approach for Multiclass segmentation?
The best approach would typically be an ensemble (2) of multi-class segmentation models (3).
However, the best next step would be to just see how well a multi-class model (i.e. using softmax) works ...
- 1,786
2
votes
How to express backpropagation dE/dV using matrix
I can directly enter the wjk matrix and zj matrix, but for xi, I only have to enter x1 while the xi matrix consists of x1, x2, and x3.
$W_{jk}$ is not a matrix, it's a real number whose value ...
- 54.5k
1
vote
Viewing a neural net loss function as a Gaussian Process
Neural Networks as Gaussian Processes
Consider a neural network with only one layer (i.e. no hidden layers, i.e. logistic regression): $$\operatorname{reg}: \mathbb{R}^N \to \mathbb{R}^M : \boldsymbol{...
- 1,786
3
votes
Viewing a neural net loss function as a Gaussian Process
Disclaimer: I just glimpsed at the linked articles. My answer focuses solely on Gaussian processes per se.
Your example
If you understand your example $f(x)=x^2+Y$ as a random function of $x$, then it ...
- 2,247
1
vote
Accepted
How do LLMs transform tokens into vectors?
Yes, it is exactly as you say. LMs (and machine translation models, too) start with a randomly initialized embedding matrix, which is learned via standard backpropagation. It is typically not ...
- 2,946
1
vote
Using 1-Layer Fully-Connected Neural Network to Appoximate Exponential Functions
I think your professor means that $e^x$ cannot be approximated by a single layer of ReLU globally (on the entire $\mathbb{R}$), which seems correct because whatever the output of a single layer of ...
- 181
3
votes
Having more feed forward layers as hidden layers in word2vec
Word2vec is popular because it's simple while being good enough. Because of being simple, it's fast and needs less expensive hardware to run. Yes, using one vs two layers does not seem to make much ...
Tim♦
- 127k
0
votes
Covering number of a single-layer neural network with Lipschitz Function
See this paper: https://arxiv.org/abs/1706.08498 for a solution, which introduces a covering bound on matrix-matrix products together with an inductive technique to combine covers of layers.
- 1
0
votes
Accepted
Very balanced dataset and a multiclass classification problem, no context behind the inputs. Which evaluation metric to use?
The trouble with accuracy is that your model does not predict discrete classes. The neural network outputs values on a continuum that have a (granted, weak) interpretation as the probabilities of ...
- 46.4k
0
votes
Accepted
question about mlp() in tidymodels
It's a model with a single hidden layer, plus the input and output layers, with all nodes connected from one layer to the next. That's the simplest case of multilayer perceptron (see eg Wikipedia).
...
- 28.9k
0
votes
question about mlp() in tidymodels
I agree that the explanation is confusing. The a.k.a. section refers to "perceptron model", not "multilayer perceptron model".
You can sense this from the tinymodel definition of ...
- 54.5k
0
votes
Bayesian Linear Regression on the top of deterministic neural network
There's a simple option, if you are fine with maximum-a-posteriori estimation and prediction, but don't care so much about uncertainty. I guess you don't, because it's hard to see how you'd get that ...
- 27.1k
0
votes
How to determine the no of multiplication operations in convolution operation?
Sorry, but gunes made some mistakes that I have to point out. For each output pixel, we actually have 5 x 5 x 192 multiplications. The output size is 28 x 28 x 32 (not 28 x 28 x 192). The thing you ...
14
votes
Accepted
Why must a product of symmetric random variables be symmetric?
To say that a random variable $W$ "has a symmetric distribution around zero" is saying that $W$ and $-W$ have the same distribution.
Let $X$ be another random variable and set $Y=WX.$ By ...
- 306k
2
votes
Accepted
which one between XGBoost and neural networks is more interpretable?
Both (sufficiently large) neural networks and XGBoost are not interpretable on their own (they are not "intrinsically interpretable") and are typically seen as part of the same "not ...
- 1,846
1
vote
Clarification on the connection between deep ensembles and bayesian neural networks
Bayesian neural network is a fake technology. Take any example, which shows distribution of the target and change data into multimodal, just split them into two blocks with the gap between, launch BNN ...
- 11
1
vote
What are alternatives of Gradient Descent?
(1) Bipropagation is a semi-gradient descent algorithm much faster than backpropagation. It solves the XOR problem each time and is 20 times faster than the fastest attempt of backpropagation.
(2) ...
- 11
2
votes
Accepted
How to initialize final layer to get a good starting loss?
It can be done using the bias of the final layer of the network. Here I'll show how to derive it for a balanced classification problem, but the same can be done for unbalanced problems or regression ...
- 274
0
votes
What is the origin of the autoencoder neural networks?
The first clear autoencoder presentation featuring a feedforward, multilayer neural network with a bottleneck layer was presented by Kramer in 1991 (full text at https://people.engr.tamu.edu/rgutier/...
-1
votes
How to initialize final layer to get a good starting loss?
You could use the Xavier initialization described in this paper. It allows the output of the classification layer (fully connected + softmax activation function) to be close to a uniform distribution, ...
0
votes
How to calculate the decay rate given an initial learning rate and final learning rate for schedulers when training neural networks?
I ended up figuring it out.
For the exponential decay, it was easier than I thought, as the formula for the decay is
...
1
vote
Accepted
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
It is actually the same cross-entropy as with current language models; only the notation is different. It is well illustrated in Figure 1 of the paper:
Function $f$ already returns the $i$-th index ...
- 2,946
1
vote
Accepted
How does masking work to make neural networks use varying input lengths
Masking is not enforcing weights to be zero, masking enforces network inputs and intermediate hidden states to be zero. Setting the inputs to be zero at the begging is not enough; many places in a NN ...
- 2,946
Top 50 recent answers are included