1
$\begingroup$

I've read in some posts that the absolute function is not differentiable. However, it is not differentiable only at 0. Can I not check the value at each training point to calculate the derivative. Only if the error happens to be 0 will it create a problem, in which case I could ignore the point. Simplex is the preferred way, probably because it is more efficient, but I just want to understand that is efficiency the only reason why we might not want to use gradient descent or is there some other issue as well.

$\endgroup$
5
$\begingroup$

This difficulty can be addressed using subgradients. A subgradient of a convex function $f$ at some point $x$ is any value $v$ that satisfies $f(y)-f(x)\ge v\cdot (y-x)$ for all $y$ in the domain. Note that if $f$ is differentiable at $x$ then the subgradient is just the gradient.

The subgradient method is very similar to gradient descent in concept; update the current solution in the direction of the subgradient. That is, take $x^{\{k+1\}}=x^{\{k\}}-\alpha_k v^{\{k\}}$, where $v^{\{k\}}$ is a subgradient at $x^{\{k\}}$ and $\alpha_k$ is the step size at iteration k.

The subgradient of $f(x)=|x|$ is $v(x)=\left\lbrace \begin{array}{cc} 1, & x>0 \\ -1, & x<0 \\ [-1,1], &x=0 \end{array}\right.$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! so basically it can be solved, using sub gradient instead of gradient. However, I see you mention sub gradient has 2 values at x = 0 for your example, what value would the algorithm pick in this case? $\endgroup$ – racerX Nov 14 '17 at 6:47
  • $\begingroup$ The subgradient of $|x|$ at zero is not two values, it's the interval from -1 to 1 (infinite values). The algorithm need only pick one (e.g. 0). $\endgroup$ – Kyle Nov 14 '17 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.