441k views

### What is the trade-off between batch size and number of iterations to train a neural network?

When training a neural network, what difference does it make to set: batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$ where $ab = cd$? To ...
92k views

### How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
21k views

### How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
9k views

### Backpropagation algorithm and error in hidden layer

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
10k views

### Why is gradient descent inefficient for large data set?

Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set. What is it with ...
22k views

I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct. My understanding of the difference between ...
1k views

### Looking for book recommendations for numerical optimization

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning? and realised that I would like to learn a lot more about numerical optimization....
4k views

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
1 vote
1k views

### How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
1 vote
66 views

Suppose our hypothesis space is $$\mathcal{H}=\{f:f(x)=f_\theta (x), \theta\in \Theta\},$$ where $\theta$ is the trainable parameter. Suppose we have a dataset $\{x_i,y_i\}_{i=1}^N.$ In the notes from ...
1 vote
In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...