170k views

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\... 124 votes 7 answers 91k views ### Why use gradient descent for linear regression, when a closed-form math solution is available? I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis. h(x) = B0 + B1X why we ... 80 votes 6 answers 18k views ### Optimization when Cost Function Slow to Evaluate Gradient descent and many other methods are useful for finding local minima in cost functions. They can be efficient when the cost function can be evaluated quickly at each point, whether numerically ... 14 votes 5 answers 10k views ### Why is gradient descent inefficient for large data set? Let's say our data set contains 1 million examples, i.e.,$x_1, \ldots, x_{10^6}$, and we wish to use gradient descent to perform a logistic or linear regression on these data set. What is it with ... 18 votes 4 answers 11k views ### How to run linear regression in a parallel/distributed way for big data setting? I am working on a very large linear regression problem, with data size so large that they have to be stored on a cluster of machines. It will be way too big to aggregate all the samples into one ... 16 votes 5 answers 3k views ### For neural networks, is mini-batching done purely because of memory constraints? I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update). Is the reason purely that with big datasets, it requires big computing ... 9 votes 3 answers 14k views ### Stochastic gradient descent for regularized logistic regression At 8:30 of this video Andrew Ng mentions that the cost function for stochastic gradient descent (for a single observation) for logistic regression is$-y_i \log h_w(x_i) - (1 - y_i) \log h_w(1 - x_i) ...
22k views

I know this question is redundant and has been answered here but I still want to understand it from my point of view to make sure if my terms are correct. My understanding of the difference between ...
4k views

### Why one epoch for stochastic gradient descent (SGD) is much better than one iteration for gradient decent (GD)?

Calculating gradient needs to sum over all the data points. So, SGD can be viewed as "using one data point to weakly approximate the gradient" to save time. Intuitively, I would think One epoch for ...
3k views

I was learning the optimization part in deep learning. Let's take linear regression as a simple example. Let $m$ be the total number of data points in the training set $(X,y)$ and $n$ is the number ...
3k views

### How to identify support vectors in SGD svm?

I am using SGD svm from scikit learn. I find that unlike SVC who has support_ as a member of the model to store the index of the support vectors, SGDClassifier only gives me the weight of the decision ...
1 vote
218 views

### Performing Linear Regression using Stochastic Gradient Descent, by batches

I am presented with a data set, where I am supposed to perform linear regression on this using SGD. My first instinct would be to train each data point there is until I reach the last one. Only then ...
1 vote
168 views

### Backpropagation not incremental

Why is backpropagation not incremental? The definition of incremental would be to be able to update the weights with every single new data point. However, in stochastic gradient descent with a ...
1 vote