When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)? I was reading the paper MAML and they say:

This approximation removes the need for computing Hessian-vector products in an additional backward pass

So when doing the derivative of the derivative to update parameters of the meta-learning, do we use these hessian-vector products or Hessians? I am confused which one are used.
 A: This is a non-exhaustive answer. My guess would be, that a direct contact to the author(s) might be a lot better suited to discuss the issue.
That being said, my take on the section about classification is that the authors try to evaluate the cost-vs-benefit of computing the second derivatives (the Hessian). They actually argue that the Hessian is close to zero, suggesting a linear model. Whether this is a general feature of the MAML, or just of a particular choice I can't judge.
Not considering the particular properties of mentioned MAML, the evaluation of the second order derivatives possibly presents a quick way to train MLPs. This type of description is part of the Neural Network literature, like Bishops "Neural Networks for Pattern Recognition" from 1996. There are several papers on this as well, from Bishop, as well as from LeCun. I can't remember which one(s) exactly, but they are listed on their homepages.
So in general there is no need to calculate the Hessian (or any product). It depends on the type of network and the type of learning. To my knowledge, training of classification networks benefit most from this approach. In certain cases, training can happen in a single step (e.g.: Bishop, "Exact Calculation of the Hessian Matrix for the Multi-Layer Perceptron", 1992); it might just be expensive in terms of computation.
A: It's because the Hessian is of size $P^2$, which is absolutely ridiculously large when $P$ is the number of parameters in a neural net. Somewhere in the autodiff, something like $\mathbf{H}\mathbf{v}$ is being calculated, where $\mathbf{H}$ is the Hessian and $\mathbf{v}$ some vector. Notice that the result, $\mathbf{H}\mathbf{v}$, is only of size $P$. If we can directly compute that without ever actually forming the Hessian $\mathbf{H}$, we can still do the computation. This is the utility of directly defining a function which returns $\mathbf{H}\mathbf{v}$, or in other words, a utility of thinking of matrices as linear operators.
So to address the body of your question, we are using the Hessian mathematically, we just don't need to compute and store the whole Hessian
on a computer (which, for reference, is 32 terabytes when we have a million weight network).
