0
$\begingroup$

I have been studying AD for these days and I think I understand how it works, but all functions for which AD has been applied in the lectures I've studied are elementary in the mathematical sense, I mean, with a formula with elementary functions.

That's why I want to ask you an example of an aplication of AD to a function like a black box or if you can explain to me the way experts in the field use AD in real problems (using machine learning, for example).

Thank you very much.

$\endgroup$
2
  • 1
    $\begingroup$ If you don't know the function, you cannot apply automatic differentiation to it. In real world machine learning scenarios, though, we always know the functions that are being automatically differentiated. $\endgroup$ Feb 10, 2020 at 10:39
  • $\begingroup$ @JanKukacka: Your comment seems like a reasonable answer for me. (and I agree with it.) $\endgroup$
    – usεr11852
    Feb 10, 2020 at 13:20

2 Answers 2

1
$\begingroup$

You can't do AD without a functional form because you can't take the derivative of something that you don't know. You can approximate derivatives numerically by sampling the function after making small changes to the inputs, which is a technique used in curve fitting.

People have managed to apply AD to some non-differentiable functions (like max pooling). In max pooling, the derivative ends up being just the derivative of the max position. In reality, changing the inputs might effect which position max pooling would select, and so the function is not differentiable.

$\endgroup$
0
$\begingroup$

I'm not familiar with automatic differentiation for blackbox functions. In machine learning, the loss is typically elementary, or close to it. For example, a simple feedforward neural network consists of matrix multiplications (multiplication is certainly elementary), sigmoid (which is elementary), and perhaps you have some cross-entropy loss, which is also elementary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.