Linked Questions

384 votes
5 answers

What is the trade-off between batch size and number of iterations to train a neural network?

When training a neural network, what difference does it make to set: batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$ where $ ab = cd $? To ...
Franck Dernoncourt's user avatar
55 votes
1 answer

How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
Simon Kuang's user avatar
  • 2,101
33 votes
3 answers

How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
Alina's user avatar
  • 1,155
24 votes
3 answers

Backpropagation algorithm and error in hidden layer

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
HIGGINS's user avatar
  • 509
14 votes
5 answers

Why is gradient descent inefficient for large data set?

Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set. What is it with ...
Fraïssé's user avatar
  • 1,430
7 votes
1 answer

Gradient Descent (GD) vs Stochastic Gradient Descent (SGD)

I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct. My understanding of the difference between ...
Steven's user avatar
  • 489
6 votes
3 answers

Looking for book recommendations for numerical optimization

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning? and realised that I would like to learn a lot more about numerical optimization....
5 votes
0 answers

Stochastic gradient descent vs mini-batch gradient descent

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
Basj's user avatar
  • 438
1 vote
1 answer

How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
bbublue's user avatar
  • 37
1 vote
1 answer

Gradient descent and epoch

Suppose our hypothesis space is $$\mathcal{H}=\{f:f(x)=f_\theta (x), \theta\in \Theta\},$$ where $\theta$ is the trainable parameter. Suppose we have a dataset $\{x_i,y_i\}_{i=1}^N.$ In the notes from ...
Sam Wong's user avatar
  • 177
1 vote
2 answers

on-line regression with 1 output [closed]

I have 12 input variables from sensor (IMU) to predict 1 output (Speed of a boat) variable. Is it possible to use regression (or something else?) in this case where it is a continuous data stream from ...
hoddy's user avatar
  • 29
2 votes
1 answer

Why a minimiser of a subset of training dataset is that of the whole training set

In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...
siegfried's user avatar
  • 330