1
$\begingroup$

I've read in some posts that the absolute function is not differentiable. However, it is not differentiable only at 0. Can I not check the value at each training point to calculate the derivative. Only if the error happens to be 0 will it create a problem, in which case I could ignore the point. Simplex is the preferred way, probably because it is more efficient, but I just want to understand that is efficiency the only reason why we might not want to use gradient descent or is there some other issue as well.

$\endgroup$
1

1 Answer 1

8
$\begingroup$

This difficulty can be addressed using subgradients. A subgradient of a convex function $f$ at some point $x$ is any value $v$ that satisfies $f(y)-f(x)\ge v\cdot (y-x)$ for all $y$ in the domain. Note that if $f$ is differentiable at $x$ then the subgradient is just the gradient.

The subgradient method is very similar to gradient descent in concept; update the current solution in the direction of the subgradient. That is, take $x^{\{k+1\}}=x^{\{k\}}-\alpha_k v^{\{k\}}$, where $v^{\{k\}}$ is a subgradient at $x^{\{k\}}$ and $\alpha_k$ is the step size at iteration k.

The subgradient of $f(x)=|x|$ is $v(x)=\left\lbrace \begin{array}{cc} 1, & x>0 \\ -1, & x<0 \\ [-1,1], &x=0 \end{array}\right.$

$\endgroup$
2
  • $\begingroup$ Thanks! so basically it can be solved, using sub gradient instead of gradient. However, I see you mention sub gradient has 2 values at x = 0 for your example, what value would the algorithm pick in this case? $\endgroup$
    – racerX
    Nov 14, 2017 at 6:47
  • 1
    $\begingroup$ The subgradient of $|x|$ at zero is not two values, it's the interval from -1 to 1 (infinite values). The algorithm need only pick one (e.g. 0). $\endgroup$
    – Kyle
    Nov 14, 2017 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.