Linked Questions

150 votes
6 answers

Batch gradient descent versus stochastic gradient descent

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\...
user20616's user avatar
  • 1,541
124 votes
7 answers

Why use gradient descent for linear regression, when a closed-form math solution is available?

I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis. h(x) = B0 + B1X why we ...
Purus's user avatar
  • 1,343
80 votes
6 answers

Optimization when Cost Function Slow to Evaluate

Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically ...
Jared Becksfort's user avatar
14 votes
5 answers

Why is gradient descent inefficient for large data set?

Let's say our data set contains 1 million examples, i.e., $x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set. What is it with ...
Fraïssé's user avatar
  • 1,460
18 votes
4 answers

How to run linear regression in a parallel/distributed way for big data setting?

I am working on a very large linear regression problem, with data size so large that they have to be stored on a cluster of machines. It will be way too big to aggregate all the samples into one ...
James Bond's user avatar
16 votes
5 answers

For neural networks, is mini-batching done purely because of memory constraints?

I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update). Is the reason purely that with big datasets, it requires big computing ...
Vladimir Belik's user avatar
9 votes
3 answers

Stochastic gradient descent for regularized logistic regression

At 8:30 of this video Andrew Ng mentions that the cost function for stochastic gradient descent (for a single observation) for logistic regression is $-y_i \log h_w(x_i) - (1 - y_i) \log h_w(1 - x_i) ...
wwl's user avatar
  • 688
7 votes
1 answer

Gradient Descent (GD) vs Stochastic Gradient Descent (SGD)

I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct. My understanding of the difference between ...
Steven's user avatar
  • 489
5 votes
1 answer

Why one epoch for stochastic gradient descent (SGD) is much better than one iteration for gradient decent (GD)?

Calculating gradient needs to sum over all the data points. So, SGD can be viewed as "using one data point to weakly approximate the gradient" to save time. Intuitively, I would think One epoch for ...
Haitao Du's user avatar
  • 36.1k
4 votes
1 answer

Stochastic Gradient Descent, Mini-Batch and Batch Gradient Descent

I was learning the optimization part in deep learning. Let's take linear regression as a simple example. Let $m$ be the total number of data points in the training set $(X,y)$ and $n$ is the number ...
KevinKim's user avatar
  • 6,819
0 votes
1 answer

How to identify support vectors in SGD svm?

I am using SGD svm from scikit learn. I find that unlike SVC who has support_ as a member of the model to store the index of the support vectors, SGDClassifier only gives me the weight of the decision ...
Wei Shi Shi's user avatar
1 vote
1 answer

Performing Linear Regression using Stochastic Gradient Descent, by batches

I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
cgo's user avatar
  • 8,777
1 vote
1 answer

Backpropagation not incremental

Why is backpropagation not incremental? The definition of incremental would be to be able to update the weights with every single new data point. However, in stochastic gradient descent with a ...
aceminer's user avatar
  • 1,043
1 vote
2 answers

Whats the horizontal and vertical axis denoting in the below SGD contour

What is that oscillation in the y axis in gradient descent contour.
KAUSHIK DEY's user avatar
1 vote
2 answers

on-line regression with 1 output [closed]

I have 12 input variables from sensor (IMU) to predict 1 output (Speed of a boat) variable. Is it possible to use regression (or something else?) in this case where it is a continuous data stream from ...
hoddy's user avatar
  • 29