The tag has no usage guidance.

213 questions
Filter by
Sorted by
Tagged with
1 vote
17 views

Stochastic gradient descent allows us to avoid the computation of full gradients at the expense of introducing a noise floor to convergence. To decrease this noise floor, SGD requires a decrease in ...
• 111
11 views

EWMA formula for SGD with momentum different than generic EWMA formula

I am currently trying to understand how SGD with momentum works, what I understand is it uses the Exponential Weighted Moving Average concept to make the updates smoother. We take weighted average of ...
• 275
11 views

Is there room for finding a more efficient hybrid optimization problem, in the context of optimization algorithms for MLE?

Recently finished my statistical modelling class, but it only briefly touched on Maximum Likelihood Estimates and I thought it was an interesting topic, so I decided to go deeper in my own time. I ...
17 views

Error term in SGD with momentum

I am reading the article "How Momentum really works" (https://distill.pub/2017/momentum/), and i am confused in one point: I am trying to derive the convergence rate for momentum from the ...
• 121
26 views

Question on the Partial Derivative of the Cross-Entropy Loss in SGD for Neural Networks

I'm currently learning about neural networks and stumbled upon a confusion related to the use of Stochastic Gradient Descent (SGD) in training. Specifically, I'm puzzled about the computation of the ...
14 views

Neural net and category separation with interaction terms

We have a stream of tabular data consiting of categorical and numerical features. One category is somehow crucial in either affecting the target, interacting with other features and sorting the data ...
57 views

Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
• 111
50 views

Simple RNN for predicting the next character [duplicate]

I implemented a simple RNN from scratch (using only the numpy library )for predicting the next characters, and I trained it on a simple text=“hello world”. It works ...
22 views

Understanding "Understanding the difficulty of training deep feedforward neural networks"

I'm following up on this question with a slightly more specific clarification I'd like to have addressed. I'm well familiar with covariance matrices as a matrix-valued generalization to random vectors ...
• 103
65 views

Clarifying the arguments of "Understanding the difficulty of training deep feedforward neural networks"

EDIT: Following on the comment of Sycorax, I am assuming that equation (4) is an "immediate consequence" of assuming the relative linearity of $f$ under the "regime" of our inputs. ...
• 103
10 views

Update rule with gradient of loss AND loss itself multiplied

I was reading a paper about Neural Holography (page 5, equation 4), where authors used simple stochastic gradient descent as optimizing method. There I have encountered following update rule: , where ...
• 101
54 views

Calculating derivative for the final layer of a neural network

I'm first learning about backpropagation in neural networks. We're doing stochastic gradient descent. The lecture provides incomplete detail on computing the derivatives for the final layer. We have ...
• 153
1 vote
38 views

Do common implementations of mini-batch gradient descent violate the i.i.d assumption needed for unbiased estimation?

When we perform mini-batch GD, we estimate the true gradient: $$\nabla L = \frac{1}{N} \sum_i \nabla L_i$$ with: $$\nabla_B L = \frac{1}{B} \sum_{i \in B} \nabla L_i$$ where $B$ is the batch size. ...
• 471
99 views

• 11
1 vote
179 views

How does SGD training error decrease in subsequent epochs with non-iid samples when it is recommended that samples in subsequent epochs be iid?

I have been reading the Deep Learning book by Ian Goodfellow and on pg. 277, they mention: It is also crucial that the minibatches be selected randomly. Computing an unbiased estimate of the expected ...
• 131
666 views

which training mode is more convenient for small datasets?

I have a regression problem to be solved using one of neural networks models, but I have a small dataset which contains 30 samples. Which training mode is more suitable for such dataset: stochastic or ...
• 153
176 views

Bias introduced when using weak shuffling

I have batch learning problem (in this particular case a neural network) where I am training my data in batches, and then repeating for a number of epochs. In Stochastic Gradient Descent, we minimise ...
144 views

About update procedure in data incremental learning

As far as I understood, the idea of data incremental learning consists of keeping the model always up to date. Suppose that we trained a model for user recognition using voice as input. Therefore, the ...
• 203
1k views

Why does ADAM optimization perform well on non-convex functions and bad on convex functions?

I'm currently trying to understand SGD and ADAM optimization, and I understand that ADAM optimization performs well on non-convex loss functions and that SGD performs well on convex loss functions (...
• 237
1 vote
258 views

Does always gradients in mini-batch SGD have to be unbiased in order to prove convergence?

I am currently reading this paper [1] and [2]. The authors state that: Our analytical results include almost all of the unbiased compression techniques. And also: (i) gradient compression must be ...
138 views

Stochastic Gradient Descent Code Check for Least Squares

I have an R-based implementation of the gradient descent and am trying to also get it to work as SGD. The function matches R's lm function when using the entire data set. But, when I sample from the ...
• 193
1 vote
247 views

Performing Linear Regression using Stochastic Gradient Descent, by batches

I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
• 9,177
817 views

Why is a 2nd order derivative optimization better for no hidden layer neural networks?

I was reading in this blog. That first order derivative SGD optimization methods are worse for neural networks without hidden layers and 2nd order is better, because that's what regression uses. Why ...
• 710
1 vote
528 views

Why does stochastic gradient descent lead us to a minimum at all?

Why do we think that stochastic gradient descent is going to find a minimum at all? I mean on each iteration SGD moves in the direction that reduces only current batch's error (SGD doesn't care about ...
• 551
713 views

The reason (and intuition) behind why stochastic gradient descent can get stuck on a local minimum

Suppose you want to find $k$ that minimises your cost function $J(k)$. We may want to apply batch gradient descent or stochastic gradient descent. Let's deliberately initialise $k$ with the same ...
• 551