471k views

### What is the trade-off between batch size and number of iterations to train a neural network?

When training a neural network, what difference does it make to set: batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$ where $ab = cd$? To ...
• 47.4k
94k views

### How large should the batch size be for stochastic gradient descent?

I understand that stochastic gradient descent may be used to optimize a neural network using backpropagation by updating each iteration with a different sample of the training dataset. How large ...
• 2,121
22k views

### How could stochastic gradient descent save time compared to standard gradient descent?

Standard Gradient Descent would compute gradient for the entire training dataset. ...
• 1,175
11k views

### Backpropagation algorithm and error in hidden layer

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP). The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of ...
• 509
11k views

### Why is gradient descent inefficient for large data set?

Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set. What is it with ...
• 1,590
23k views

I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct. My understanding of the difference between ...
• 499
2k views

### Looking for book recommendations for numerical optimization

I was reading the answers and comments to this question: Why is Newton's method not widely used in machine learning? and realised that I would like to learn a lot more about numerical optimization....
4k views

Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-...
• 612
1 vote
1k views

### How does the batch size affect the Stochastic Gradient Descent optimizer? (Example using Keras)

First of all, I know that there are lots of questions and answers about the topic throughout the site $-$ such as here, here or here (and I've probably read them all). However, I am still confused. ...
• 37
1 vote
83 views

Suppose our hypothesis space is $$\mathcal{H}=\{f:f(x)=f_\theta (x), \theta\in \Theta\},$$ where $\theta$ is the trainable parameter. Suppose we have a dataset $\{x_i,y_i\}_{i=1}^N.$ In the notes from ...
• 177
1 vote
68 views

### on-line regression with 1 output [closed]

I have 12 input variables from sensor (IMU) to predict 1 output (Speed of a boat) variable. Is it possible to use regression (or something else?) in this case where it is a continuous data stream from ...
• 29
In section 3.3 of Bottou et al (2018), under the 'intuitive motivation' paragraph, the authors claim that 'a minimiser of empirical risk for the larger set $S$ is clearly given by a minimiser for the ...