# Inconsistency in unit on gradient descent equation

The update algorithm for gradient descent is

$$\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$$

$\frac{\partial}{\partial \theta_j}J(\theta)$ have the unit of cost per unit $\theta$ but $\theta_j$ have the unit of $\theta$ and constant $\alpha$ is dimentionless, why is it valid that we can use subtraction for two element with different units?

• What is the basis for your assertions about the units in your question? You might want to examine each statement carefully. Sep 25, 2016 at 5:45
• fair point,I think the justification for the units of derivative is self explanatory using Leibniz's definition, and the unit of $\theta_j$ is trivial, the only unit I am not sure is $\alpha$, are we suggesting that $\alpha$ have a unit of $\frac{\theta^2}{Cost}$ , if so is there any intuition behind that? Are we just arbitrary assigning units to a constant so that the dimension matches? Sep 25, 2016 at 5:53
• Take a look at this, it might help solidify the notion you're currently considering -- timvieira.github.io/blog/post/2016/05/27/… Sep 25, 2016 at 6:30

As was concluded in the discussion in comments, dimensional analysis would necessitate that the relevant component of $\alpha$ is in fact in the units necessary to make
$$\alpha_j \frac{\partial}{\partial \theta_j}J(\theta)$$
have the same units as $\theta_j$
• Thanks for the answer, but I am slightly confused, using $\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$ with $\alpha$ compensating for dimension correctness is quite unintuitive, why not just use $\theta_j = \theta_j - \alpha( \frac{\partial}{\partial \theta_j}J(\theta))^{-1}$ where $\alpha$ has the unit of $J(\theta)$ Sep 25, 2016 at 23:02
• Because it won't head you in the right direction! The point is to go downhill as fast as possible, not to satisfy your personal aesthetic sense of what makes the units of the relevant component of $\alpha$ nice. Sep 26, 2016 at 0:00