Linked Questions
15 questions linked to/from How could stochastic gradient descent save time compared to standard gradient descent?
150
votes
6
answers
170k
views
Batch gradient descent versus stochastic gradient descent
Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
- 1,541
124
votes
7
answers
91k
views
Why use gradient descent for linear regression, when a closed-form math solution is available?
I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis.
h(x) = B0 + B1X
why we ...
- 1,343
80
votes
6
answers
18k
views
Optimization when Cost Function Slow to Evaluate
Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically ...
- 1,023
14
votes
5
answers
10k
views
Why is gradient descent inefficient for large data set?
Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set.
What is it with ...
- 1,460
18
votes
4
answers
11k
views
How to run linear regression in a parallel/distributed way for big data setting?
I am working on a very large linear regression problem, with data size so large that they have to be stored on a cluster of machines. It will be way too big to aggregate all the samples into one ...
- 381
16
votes
5
answers
3k
views
For neural networks, is mini-batching done purely because of memory constraints?
I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing ...
- 1,531
9
votes
3
answers
14k
views
Stochastic gradient descent for regularized logistic regression
At 8:30 of this video Andrew Ng mentions that the cost function for stochastic gradient descent (for a single observation) for logistic regression is
$-y_i \log h_w(x_i) - (1 - y_i) \log h_w(1 - x_i) ...
- 688
7
votes
1
answer
22k
views
Gradient Descent (GD) vs Stochastic Gradient Descent (SGD)
I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct.
My understanding of the difference between ...
- 489
5
votes
1
answer
4k
views
Why one epoch for stochastic gradient descent (SGD) is much better than one iteration for gradient decent (GD)?
Calculating gradient needs to sum over all the data points. So, SGD can be viewed as "using one data point to weakly approximate the gradient" to save time.
Intuitively, I would think One epoch for ...
- 36.1k
4
votes
1
answer
3k
views
Stochastic Gradient Descent, Mini-Batch and Batch Gradient Descent
I was learning the optimization part in deep learning.
Let's take linear regression as a simple example. Let $m$ be the total number of data points in the training set $(X,y)$ and $n$ is the number ...
- 6,819
0
votes
1
answer
3k
views
How to identify support vectors in SGD svm?
I am using SGD svm from scikit learn. I find that unlike SVC who has support_ as a member of the model to store the index of the support vectors, SGDClassifier only gives me the weight of the decision ...
1
vote
1
answer
218
views
Performing Linear Regression using Stochastic Gradient Descent, by batches
I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
- 8,777
1
vote
1
answer
168
views
Backpropagation not incremental
Why is backpropagation not incremental?
The definition of incremental would be to be able to update the weights with every single new data point.
However, in stochastic gradient descent with a ...
- 1,043
1
vote
2
answers
97
views
Whats the horizontal and vertical axis denoting in the below SGD contour
What is that oscillation in the y axis in gradient descent contour.
- 73
1
vote
2
answers
67
views
on-line regression with 1 output [closed]
I have 12 input variables from sensor (IMU) to predict 1 output (Speed of a boat) variable. Is it possible to use regression (or something else?) in this case where it is a continuous data stream from ...
- 29