Showing the existence of uncountable regression function estimates with perfect fit to finite or infinite samples While I understand by intuition that there can be infinitely many regression functions (of different shapes) that can go through the sample points in a plane (considering only one predictor and one outcome) producing a perfect fit to the data, I am wondering if there is any formal way of showing that rigorously. First, I'd like to define a set that contains such regression functions. Then I'd like to show that this set can have an infinite number of such functions.  
 A: Reduce the problem to one where the responses are all zero and construct a family of nice functions having this set of common zeros.
Details follow.

Let the distinct regressor values be $R = \{x_1, x_2, \ldots, x_d\}.$  Let $p$ be any nonzero real-valued function defined for all real numbers that is zero on $R.$  One such $p$--to show they exist--is
$$p_R(x) = \prod_{i=1}^d (x-x_i).$$
If there is just one $y_i$ associated with each $x_i,$ then you can find at least one function agreeing with every $(x_i,y_i):$ for instance, connect the points from left to right with line segments and extend horizontal rays away from the two extreme points.  The result is the graph of a piecewise linear function $f$ passing through all the points.  That is, $f(x_i) = y_i$ for each $i.$
It is almost as easy to find a polynomial $f$ passing through all the points.
The family
$$\{\lambda p + f \mid \lambda \in \mathbb R\}$$
consists of functions also passing through all points because
$$(\lambda p + f)(x_i) = \lambda p(x_i) + f(x_i) = \lambda(0) + y_i = y_i$$
for all $i.$  These functions are in one-to-one correspondence with $\lambda\in\mathbb R$ (and therefore are uncountable) because if $\lambda p + f = \mu p + f,$ then their difference $(\lambda - \mu)p$ is identically zero, which (because $p$ is nonzero) happens only when $\lambda=\mu.$
Notice that when $f$ is a polynomial, every function in this family is "nice" in most practical and mathematical senses of the word: they are all continuous, differentiable, infinitely differentiable, analytic, and algebraic.

I notice a mention of "infinite samples" in the title.  In case the regressors are infinite in number, you must require that they have no accumulation points (for otherwise the function can get rather wild near any such point).  A standard way to construct a function that has zeros at such a set of regressors $R=\{x_1,x_2, \ldots, x_n, \ldots\}$ (which must be countable and not include $0$) is to form
$$q_R(x) = \sum_{i=1}^\infty b_i\left(\frac{1}{x-x_i}+\frac{1}{x_i}\right)$$
for a sequence $(b_i)$ that decreases sufficiently rapidly to make this sum converge outside $R,$ and then define $p_R(x) = 1/q_R(x)$ for $x\notin R$ and $p_R(x)=0$ otherwise (Whittaker and Watson section 7.4).  This will be analytic (throughout the Complex plane) and therefore is continuous and even infinitely differentiable.
Reference
E. T. Whittaker and G. N. Watson, A Course of Modern Analysis, Fourth Edition 1927 (reprinted 1973).  Cambridge University Press.
