What is the continuous depth concept in Neural ODE paper by David Duvenaud?

Question

I read the ever so celebrated -https://arxiv.org/abs/1806.07366 paper. I am still confused. Does the concept of continuous depth neural network correspond to the fact that the ODEint or the integrator block takes into account the value of the derivative to decide the step size and then performs an "integration"? Or does it refer to something else?

Answer 1

I wouldn't read into the term "continuous depth" too much. It's just that since the ODE allows you to evaluate the neural network at any layer (for example we could compute $h(\pi)$ to obtain the value of the network at $\pi = 3.14\ldots$, the concept of depth and number of layers is not meaningful. You could say there are an infinite number of layers, but that's not a helpful description.

The authors use NFE (number of function evaluations) as a proxy for "depth" because just as we expect model expressivity and complexity to increase with depth in a typical network, we expect more powerful neural ODEs to require a larger NFE to solve. I think this is what you were getting at. While this definition of "depth" is useful for comparing model capacities and computational efficiency, it's very distinct concept from my usual conception of network depth.