Return to Question

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\theta}(x_{(i)}) = \theta_0+\theta_{1}x_{(i)1} + \cdots +\theta_{n}x_{(i)n}$. We need to find the parameters $\mathbf{\theta}$ that minimize the "distance" between $y_{(i)}$ and $h_{\theta}(x_{(i)})$. Let $$J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (y_{(i)}-h_{\theta}(x_{(i)})^{2}$$

Then we want to find $\theta$ that minimizes $J(\theta)$. In gradient descent we initialize each parameter and perform the following update: $$\theta_j := \theta_j-\alpha \frac{\partial}{\partial \theta_{j}} J(\theta)$$

What is the key difference between batch gradient descent and stochastic gradient descent?

Both use the above update rule. But is one better than the other?

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\theta}(x_{(i)}) = \theta_0+\theta_{1}x_{(i)1} + \cdots +\theta_{n}x_{(i)n}$. We need to find the parameters $\mathbf{\theta}$ that minimize the "distance" between $y_{(i)}$ and $h_{\theta}(x_{(i)})$. Let $$J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (y_{(i)}-h_{\theta}(x_{(i)}))^{2}$$

Then we want to find $\theta$ that minimizes $J(\theta)$. In gradient descent we initialize each parameter and perform the following update: $$\theta_j := \theta_j-\alpha \frac{\partial J(\theta)}{\partial \theta_{j}} $$

What is the key difference between batch gradient descent and stochastic gradient descent?

Both use the above update rule. But is one better than the other?

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\theta}(x_{(i)}) = \theta_0+\theta_{1}x_{(i)1} + \cdots +\theta_{n}x_{(i)n}$. We need to find the parameters $\mathbf{\theta}$ that minimize the "distance" between $y_{(i)}$ and $h_{\theta}(x_{(i)})$. Let $$J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (y_{(i)}-h_{\theta}(x_{(i)})^{2}$$

Then we want to find $\theta$ that minimizes $J(\theta)$. In gradient descent we initialize each parameter and perform the following update: $$\theta_j := \theta_j-\alpha \frac{\partial}{\partial \theta_{j}} J(\theta)$$

Question. What is the key difference between batch gradient descent and stochastic gradient descent?

Both use the above update rule. But is one better than the other?

Suppose we have some training set $(x_{(i)}, y_{(i)})$ for $i = 1, \dots, m$. Also suppose we run some type of supervised learning algorithm on the training set. Hypotheses are represented as $h_{\theta}(x_{(i)}) = \theta_0+\theta_{1}x_{(i)1} + \cdots +\theta_{n}x_{(i)n}$. We need to find the parameters $\mathbf{\theta}$ that minimize the "distance" between $y_{(i)}$ and $h_{\theta}(x_{(i)})$. Let $$J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (y_{(i)}-h_{\theta}(x_{(i)})^{2}$$

Then we want to find $\theta$ that minimizes $J(\theta)$. In gradient descent we initialize each parameter and perform the following update: $$\theta_j := \theta_j-\alpha \frac{\partial}{\partial \theta_{j}} J(\theta)$$

What is the key difference between batch gradient descent and stochastic gradient descent?

Both use the above update rule. But is one better than the other?