Linked Questions
12 questions linked to/from Batch gradient descent versus stochastic gradient descent
391
votes
5
answers
471k
views
What is the trade-off between batch size and number of iterations to train a neural network?
When training a neural network, what difference does it make to set:
batch size to $a$ and number of iterations to $b$
vs. batch size to $c$ and number of iterations to $d$
where $ ab = cd $?
To ...
- 47.4k
55
votes
1
answer
94k
views
How large should the batch size be for stochastic gradient descent?
I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
- 2,121
33
votes
3
answers
22k
views
How could stochastic gradient descent save time compared to standard gradient descent?
Standard Gradient Descent would compute gradient for the entire training dataset.
...
- 1,175
24
votes
3
answers
11k
views
Backpropagation algorithm and error in hidden layer
I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP).
The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
- 509
14
votes
5
answers
11k
views
Why is gradient descent inefficient for large data set?
Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set.
What is it with ...
- 1,590
7
votes
1
answer
23k
views
Gradient Descent (GD) vs Stochastic Gradient Descent (SGD)
I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct.
My understanding of the difference between ...
- 499
6
votes
3
answers
2k
views
Looking for book recommendations for numerical optimization
I was reading the answers and comments to this question:
Why is Newton's method not widely used in machine learning?
and realised that I would like to learn a lot more about numerical optimization....
5
votes
0
answers
4k
views
Stochastic gradient descent vs mini-batch gradient descent
Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
- 612
1
vote
1
answer
1k
views
How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)
First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
- 37
1
vote
1
answer
83
views
Gradient descent and epoch
Suppose our hypothesis space is $$\mathcal{H}=\{f:f(x)=f_\theta (x), \theta\in \Theta\},$$ where $\theta$ is the trainable parameter.
Suppose we have a dataset $\{x_i,y_i\}_{i=1}^N.$
In the notes from ...
- 177
1
vote
2
answers
68
views
on-line regression with 1 output [closed]
I have 12 input variables from sensor (IMU) to predict 1 output (Speed of a boat) variable. Is it possible to use regression (or something else?) in this case where it is a continuous data stream from ...
- 29
2
votes
1
answer
35
views
Why a minimiser of a subset of training dataset is that of the whole training set
In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...
- 330