Gradient descent optimization

Question

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a scenario where weights $w_1, w_2$ are being optimized to give the minimum error, and partial derivatives are being used, does it change both $w_1$ and $w_2$ in each step or is it a combination (e.g., in few iterations only $w_1$ is changed and when $w_1$ isn't reducing the error any more, the derivative starts with $w_2$)? The application could be a linear regression model, a logistic regression model, or boosting algorithms.

Answer 1

Gradient descent updates all parameters at each step. You can see this in the update rule:

$$ w^{(t+1)}=w^{(t)} - \eta\nabla f\left(w^{(t)}\right). $$

Since the gradient of the loss function $\nabla f(w)$ is vector-valued with dimension matching that of $w$, all parameters are updated at each iteration.

The learning rate $\eta$ is a positive number that re-scales the gradient. Taking too large a step can endlessly bounce you across the loss surface with no improvement in your loss function; too small a step can mean tediously slow progress towards the optimum.

Although you could estimate linear regression parameters using gradient descent, it's not a good idea.

Likewise, there are better ways to estimate logistic regression coefficients.

Answer 2

When the optimization does occur through partial derivatives, in each turn does it change both w1 and w2 or is it a combination like in few iterations only w1 is changed and when w1 isn't reducing the error more, the derivative starts with w2 - to reach the local minima?

In each iteration, the algorithm will change all weights at the same time based on gradient vector. In fact, the gradient is a vector. The length of the gradient is as same as number of the weights in the model.

On the other hand, changing one parameter at a time did exist and it is called coordinate decent algorithm, which is a type of gradient free optimization algorithm. In practice, it may not work as well as gradient based algorithm.

Here is an interesting answer on gradient free algorithm

Is it possible to train a neural network without backpropagation?

Answer 3

Gradient decent is applied to both w1 and w2 for each iteration. During each iteration, the parameters updated according to the gradients. They would likely have different partial derivative.

Check here.

Answer 4

The aim of gradient descent is to minimize the cost function. This minimization is achieved by adjusting weights, for your case w1 and w2. In general there could be n such weights.

Gradient descent is done in the following way:

1. initialize weights randomly.
2. compute the cost function and gradient with initialized weights.
3. update weigths: It might happen that the gradient is O for some weights, in that case those weights doesn't show any change after updating. for example: Let say gradient is [1,0] the W2 will remain unchanged.
4. check the cost function with updated weights, if the decrement is acceptable enough continue the iterations else terminate.

while updating weights which weight ( W1 or W2) gets changed is entirely decided by gradient. All the weights get updated ( some weights might not change based on gradient).