# Gradient descent optimization

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $$\hat y-y$$. In a scenario where weights $$w_1, w_2$$ are being optimized to give the minimum error, and partial derivatives are being used, does it change both $$w_1$$ and $$w_2$$ in each step or is it a combination (e.g., in few iterations only $$w_1$$ is changed and when $$w_1$$ isn't reducing the error any more, the derivative starts with $$w_2$$)? The application could be a linear regression model, a logistic regression model, or boosting algorithms.

## 4 Answers

Gradient descent updates all parameters at each step. You can see this in the update rule:

$$w^{(t+1)}=w^{(t)} - \eta\nabla f\left(w^{(t)}\right).$$

Since the gradient of the loss function $$\nabla f(w)$$ is vector-valued with dimension matching that of $$w$$, all parameters are updated at each iteration.

The learning rate $$\eta$$ is a positive number that re-scales the gradient. Taking too large a step can endlessly bounce you across the loss surface with no improvement in your loss function; too small a step can mean tediously slow progress towards the optimum.

Although you could estimate linear regression parameters using gradient descent, it's not a good idea.

Likewise, there are better ways to estimate logistic regression coefficients.

• So the algorithm may try different combinations like increase w1 , decrease w2 based on the direction from partial derivative to reach local minima and just to confirm the algorithm will not necessarily give the global minima always?
– Pb89
Oct 22, 2018 at 3:34
• and does the partial derivative also help to explain how much increase or decrease has to be done to w1 and w2 or that is done by learning rate/shrinkage while partial derivative only provides direction of descent?
– Pb89
Oct 22, 2018 at 3:37
• The gradient is a vector, so it gives a direction and a magnitude. A vector can be arbitrarily rescaled by a positive scalar and it will have the same direction, but the rescaling will change its magnitude.
– Sycorax
Oct 22, 2018 at 3:47
• If magnitude is also given by the gradient then what is the role of shrinkage or learning rate?
– Pb89
Oct 22, 2018 at 3:49
• The learning rate rescales the gradient. Suppose $\nabla f(x)$ has a large norm (length). Taking a large step will move you to a distant part of the loss surface (jumping from one mountain to another). The core justification of gradient descent is that it's a linear approximation in the vicinity of $w^{(t)}$. That approximation is always inexact, but it's probably worse the farther away you move -- hence, you want to take small steps, so you use some small $\eta$, where 'small' is entirely problem-specific.
– Sycorax
Oct 22, 2018 at 3:52

When the optimization does occur through partial derivatives, in each turn does it change both w1 and w2 or is it a combination like in few iterations only w1 is changed and when w1 isn't reducing the error more, the derivative starts with w2 - to reach the local minima?

In each iteration, the algorithm will change all weights at the same time based on gradient vector. In fact, the gradient is a vector. The length of the gradient is as same as number of the weights in the model.

On the other hand, changing one parameter at a time did exist and it is called coordinate decent algorithm, which is a type of gradient free optimization algorithm. In practice, it may not work as well as gradient based algorithm.

Here is an interesting answer on gradient free algorithm

Is it possible to train a neural network without backpropagation?

Gradient decent is applied to both w1 and w2 for each iteration. During each iteration, the parameters updated according to the gradients. They would likely have different partial derivative.

Check here.

The aim of gradient descent is to minimize the cost function. This minimization is achieved by adjusting weights, for your case w1 and w2. In general there could be n such weights.

Gradient descent is done in the following way:

1. initialize weights randomly.
2. compute the cost function and gradient with initialized weights.
3. update weigths: It might happen that the gradient is O for some weights, in that case those weights doesn't show any change after updating. for example: Let say gradient is [1,0] the W2 will remain unchanged.
4. check the cost function with updated weights, if the decrement is acceptable enough continue the iterations else terminate.

while updating weights which weight ( W1 or W2) gets changed is entirely decided by gradient. All the weights get updated ( some weights might not change based on gradient).

• "if the decrement is acceptable enough continue the iterations else terminate", is there a default value which is applied in packages of python ( sklearn) or R packages such as caret? It can be user specified only in a manually created gradient descent function?
– Pb89
Oct 22, 2018 at 7:17