What does interpolating the training set actually mean? I just read this article: Understanding Deep Learning (Still) Requires Rethinking Generalization
In section 6.1 I stumbled upon the following sentence

Specifically, in the overparameterized regime where the model capacity
greatly exceeds the training set size, fitting all the training
examples (i.e., interpolating the training set), including noisy ones,
is not necessarily at odds with generalization.

I do not fully understand the term "interpolating" in the context of fitting training data. Why do we speak of "interpolation" in this context? What does the term exactly mean here? Is there any other term that can be used instead?
In my understanding interpolation means the prediction within the training domain for some novel input that was not part of the training set.
 A: Apart from literal meaning of interpolation, this is related to  something called deep learning models totally memorize the training data. Hence, both interpolating and memorisation in this paper/context means zero training loss but still not overfitting on the test set. Hence the curious phenomena, that normally we would call an overfitting (overtraining actually), which is not resolved yet.
A: I would add, the quote actually contains the definition, "fitting all the training examples... including noisy ones". So the training loss is zero. The comment "including noisy ones" implies the the data is generated by a process say
$y= f(x) + \epsilon$
Where $\epsilon$ represents noise. By fitting the model such that $y=f(x)$ for every training example even when $\epsilon$ is non-zero you are interpolating.
A: Your question already got two nice answers, but I feel that some more context is needed.
First, we are talking here about overparametrized models and the double descent phenomenon. By overparametrized models we mean such that have way more parameters than datapoints. For example, Neal (2019), and Neal et al (2018) trained a network with hundreds of thousands of parameters for a sample of 100 MNIST images. The discussed models are so large, that they would be unreasonable for any practical applications. Because they are so large, they are able to fully memorize the training data. Before the double descent phenomenon attracted more attention in the machine learning community, memorizing the training data was assumed to lead to overfitting and poor generalization in general.
As already mentioned by @jcken, if a model has a huge number of parameters, it can easily fit a function to the data such that it "connects all the dots" and at prediction time just interpolates between the points. I'll repeat myself, but until recently we would assume that this would lead to overfitting and poor performance. With the insanely huge models, this doesn't have to be the case. The models would still interpolate, but the function would be so flexible that it won't hurt the test set performance.
To understand it better, consider the lottery ticket hypothesis. Loosely speaking, it says that if you randomly initialize and train a big machine learning model (deep network), this network would contain a smaller sub-network, the "lottery ticket", such that you could prune the big network while keeping the performance guarantees. The image below (taken from the linked post), illustrates such pruning. Having a huge number of parameters is like buying piles of lottery tickets, the more you have, the higher your chance of winning. In such a case, you can find a lottery ticket model that interpolates between the datapoints but also generalizes.

Another way to think about it is to consider a neural network as a kind of ensemble model. Every neural network has a pre-ultimate layer (image below, adapted from this), that you can think of as a collection of intermediate representations of your problem. The outputs of this layer are then aggregated (usually using a dense layer) for making the final prediction. This is like ensembling many smaller models. Again, if the smaller models memorized the data, even if each would overfit, by aggregating them, the effects would hopefully cancel out.

All the machine learning algorithms kind of interpolate between the datapoints, but if you more parameters than data, you would literally memorize the data and interpolate between them.
A: In layman's terms, an interpolator will literally 'join the dots'.
Here's a simple graphical summary of what interpolation can do and why it can be awful. I'd like to stress that interpolation does play a useful role in statistics/ml but should be used carefully. The  black dots are training data and the red crossed are a similar dataset drawn form the same data generating process - they can be though of as a test set. 
We can see the in the left hand plot fits okay to both the training and test data. On the left I just used linear regression to fit a line to the data (just $2$ parameters). The curve in the right hand plot perfectly predicts the training set but looks nothing like the test set. I'd used an $11^{th}$ order polynomial (plus intercept) to fit the interpolator. Additionally, on the test set, the linear fit gives $MSE = 23.8$, the interpolator gives $MSE = 10350842349$ -- not good!
