The Chen et. al paper Neural ODE (https://arxiv.org/pdf/1806.07366.pdf) uses the adjoint method to take derivatives of solutions generated by an ODE solver with respect to neural network parameters θ. A scalar-valued loss function L() is defined, whose input is the result of an ODE solver. Then the authors present the following statement:
"To optimize L, we require gradients with respect to θ. The first step is to determining how the gradient of the loss depends on the hidden state z(t) at each instant. This quantity is called the adjoint a(t) = ∂L/∂z(t)."
In the literature that I have found, the adjoint method is used as an alternative for computing an inner product between vectors where a function (or its tangent linear approximation) is involved. For example, if A is a linear operator mapping between vector spaces $\mathbb{R}^n$ and $\mathbb{R}^m$, $u \in \mathbb{R}^n$ and $v \in \mathbb{R}^m$:
$$ <v, Au> = <A^*v, u> $$
where $A^*$ is the adjoint of $A$ and $< , >$ designates an inner product.
I cannot understand why ∂L/∂z(t) is the adjoint of ∂L/∂θ (I'm guessing?) in the neural ODE paper. ∂L/∂z(t) and ∂L/∂θ do not seem to provide mappings between the same vector spaces so I cannot see why it can be inferred that they are adjoints.
Any formal proof or intuition that explains how adjoints are defined in this work would be greatly appreciated.
