Inconsistency in unit on gradient descent equation

The update algorithm for gradient descent is

$$\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$$

$\frac{\partial}{\partial \theta_j}J(\theta)$ have the unit of cost per unit $\theta$ but $\theta_j$ have the unit of $\theta$ and constant $\alpha$ is dimentionless, why is it valid that we can use subtraction for two element with different units?

• What is the basis for your assertions about the units in your question? You might want to examine each statement carefully. – Glen_b Sep 25 '16 at 5:45
• fair point,I think the justification for the units of derivative is self explanatory using Leibniz's definition, and the unit of $\theta_j$ is trivial, the only unit I am not sure is $\alpha$, are we suggesting that $\alpha$ have a unit of $\frac{\theta^2}{Cost}$ , if so is there any intuition behind that? Are we just arbitrary assigning units to a constant so that the dimension matches? – user411754 Sep 25 '16 at 5:53
• Take a look at this, it might help solidify the notion you're currently considering -- timvieira.github.io/blog/post/2016/05/27/… – Glen_b Sep 25 '16 at 6:30

As was concluded in the discussion in comments, dimensional analysis would necessitate that the relevant component of $\alpha$ is in fact in the units necessary to make
$$\alpha_j \frac{\partial}{\partial \theta_j}J(\theta)$$
have the same units as $\theta_j$
• Thanks for the answer, but I am slightly confused, using $\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$ with $\alpha$ compensating for dimension correctness is quite unintuitive, why not just use $\theta_j = \theta_j - \alpha( \frac{\partial}{\partial \theta_j}J(\theta))^{-1}$ where $\alpha$ has the unit of $J(\theta)$ – user411754 Sep 25 '16 at 23:02
• Because it won't head you in the right direction! The point is to go downhill as fast as possible, not to satisfy your personal aesthetic sense of what makes the units of the relevant component of $\alpha$ nice. – Glen_b Sep 26 '16 at 0:00