3
$\begingroup$

$MAE=|y_{pred} - y_{true}|$

$\dfrac{dMAE}{dy_{pred}} = ?$

I'm trying to understand how MAE works as a loss function in neural networks using backpropogation. I know it can be used directly in some APIs - e.g. Keras - however I see tensorflow doesn't allow it (although you can manually declare it tf.abs(tf.minus(y_pred,y_true))).

My question is: How is the derivative of MAE generally calculated (I only found this which uses an apparently complex approximation) and specifically how is it computed in tensorflow (when manually declaring) and keras?

$\endgroup$
3
$\begingroup$

The , as a function of $y_{\text{pred}}$, is not differentiable at $y_{\text{pred}}=y_{\text{true}}$. Elsewhere, the derivative is $\pm 1$ by a straightforward application of the chain rule:

$$\dfrac{d\text{MAE}}{dy_{\text{pred}}} = \begin{cases} +1,\quad y_{\text{pred}}>y_{\text{true}}\\ -1,\quad y_{\text{pred}}<y_{\text{true}} \end{cases}$$

The interpretation is straightforward: if you are predicting too high ($y_{\text{pred}}>y_{\text{true}}$), then increasing $y_{\text{pred}}$ yet more by one unit will increase the MAE by an equal amount of one unit, so the gradient encourages you to reduce $y_{\text{pred}}$. And vice versa if $y_{\text{pred}}<y_{\text{true}}$.

Skimming the paper you link, it seems like they approximate the MAE by a differentiable function to avoid the "kink" at $y_{\text{pred}}=y_{\text{true}}$.

As to what specifically is implemented in TensorFlow and keras, that is off-topic here. Best to consult the documentation, the source code or any specific help community.

$\endgroup$
  • $\begingroup$ Thanks, I'm guessing implementations are programmed to default to either +1 or -1 if y_pred=y_true, I'm sure I've read somewhere that's how the Relu is normally handled. $\endgroup$ – Miss Palmer Nov 9 '17 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.