9
$\begingroup$

The update algorithm for gradient descent is

$$\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$$

$ \frac{\partial}{\partial \theta_j}J(\theta)$ have the unit of cost per unit $\theta$ but $\theta_j$ have the unit of $\theta$ and constant $\alpha$ is dimentionless, why is it valid that we can use subtraction for two element with different units?

Improve this question
$\endgroup$
3
  • $\begingroup$ What is the basis for your assertions about the units in your question? You might want to examine each statement carefully. $\endgroup$
    – Glen_b
    Sep 25, 2016 at 5:45
  • 2
    $\begingroup$ fair point,I think the justification for the units of derivative is self explanatory using Leibniz's definition, and the unit of $\theta_j$ is trivial, the only unit I am not sure is $\alpha$, are we suggesting that $\alpha$ have a unit of $\frac{\theta^2}{Cost}$ , if so is there any intuition behind that? Are we just arbitrary assigning units to a constant so that the dimension matches? $\endgroup$
    – user411754
    Sep 25, 2016 at 5:53
  • $\begingroup$ Take a look at this, it might help solidify the notion you're currently considering -- timvieira.github.io/blog/post/2016/05/27/… $\endgroup$
    – Glen_b
    Sep 25, 2016 at 6:30

2 Answers 2

Reset to default
2
$\begingroup$

As was concluded in the discussion in comments, dimensional analysis would necessitate that the relevant component of $\alpha$ is in fact in the units necessary to make

$$\alpha_j \frac{\partial}{\partial \theta_j}J(\theta)$$

have the same units as $\theta_j$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer, but I am slightly confused, using $\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$ with $\alpha$ compensating for dimension correctness is quite unintuitive, why not just use $\theta_j = \theta_j - \alpha( \frac{\partial}{\partial \theta_j}J(\theta))^{-1}$ where $\alpha$ has the unit of $J(\theta)$ $\endgroup$
    – user411754
    Sep 25, 2016 at 23:02
  • 1
    $\begingroup$ Because it won't head you in the right direction! The point is to go downhill as fast as possible, not to satisfy your personal aesthetic sense of what makes the units of the relevant component of $\alpha$ nice. $\endgroup$
    – Glen_b
    Sep 26, 2016 at 0:00
0
$\begingroup$

For the same reason that the slope of a line is not the "run", but is instead "rise" over "run", a gradient isn't a displacement in your theta-parameter space... anyone telling you otherwise is wrong. This is why the units don't match as you noted. However, the fundamental property of a gradient is that the directional derivative of a function is maximized in a direction PARALLEL to the gradient. It makes sense because gradient vector components are all the "rise" of the function divided by the "run" in each parameter (i.e. they're the slopes in each direction). If the slope of a tangent to your function (hyper-)surface is 5 times stronger in one parameter than another, you will want to move 5 times more in the strong parameter than the other. If the units are bothering you, remember that you can multiply (or divide) the gradient vector by any unit and by any scalar (like minus alpha) and you will still be parallel (or anti-parallel) to it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.