6
$\begingroup$

I read the ever so celebrated -https://arxiv.org/abs/1806.07366 paper. I am still confused. Does the concept of continuous depth neural network correspond to the fact that the ODEint or the integrator block takes into account the value of the derivative to decide the step size and then performs an "integration"? Or does it refer to something else?

$\endgroup$

1 Answer 1

6
$\begingroup$

I wouldn't read into the term "continuous depth" too much. It's just that since the ODE allows you to evaluate the neural network at any layer (for example we could compute $h(\pi)$ to obtain the value of the network at $\pi = 3.14\ldots$, the concept of depth and number of layers is not meaningful. You could say there are an infinite number of layers, but that's not a helpful description.

The authors use NFE (number of function evaluations) as a proxy for "depth" because just as we expect model expressivity and complexity to increase with depth in a typical network, we expect more powerful neural ODEs to require a larger NFE to solve. I think this is what you were getting at. While this definition of "depth" is useful for comparing model capacities and computational efficiency, it's very distinct concept from my usual conception of network depth.

$\endgroup$
3
  • $\begingroup$ Thanks for your super helpful reply. Which block do you think is doing this NFEs? Is it the integrator(Euler or RK that gets called eventually)? I do understand that there are adaptive solvers in the system. If the gradient value is very large, the "step size" is reduced for integration (and the number of evaluations or NFEs increase?). And vice-versa when the gradient is small. Is this the correct understanding? Or these NFEs is for every call made to the entire network (consisting of the neural net, integrator , loss function and the optimizer?) $\endgroup$
    – Aastha Dua
    Dec 25, 2019 at 2:01
  • $\begingroup$ NFEs are how many times the ODE solver calls the function. As im not an expert in ODE solvers, I can't say much more. $\endgroup$
    – shimao
    Dec 25, 2019 at 3:16
  • $\begingroup$ Thanks for your help @shimao! $\endgroup$
    – Aastha Dua
    Dec 25, 2019 at 4:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.