Universal Approximation Theorem and high dimension linear regression It is very well-known that neural network (NN) has the universal approximation characteristic, which seems to be one of the properties that make NN popular. 
However, I don't know how to convince myself that a high dimension linear regression model ($n/p\rightarrow 0$ where $n$ is number of samples and $p$ is the number of features) does not have such a property. Maybe a high dimensional linear regression does not guarantee such a property, but it seems to me that it can represent any function with a very high probability. 
For example, in the case with $n=1$, we should easily find a construction of the coefficient to represent any response variables. 
Similarly, if $n=5$ and $p=1000$, it does not sound hard for me to always find the coefficients to represent any response variables. But I have no idea how to formally prove anything. 
What do I miss here? Is there any similar or related theoretical work that I can look into?
For example, one can try with the following python codes:
import numpy as np
n = 5
p = 1000
X = np.random.random(size=[n, p])
y = np.random.random(size=[n, 1])
b = np.dot(np.linalg.pinv(np.dot(X.T, X)), np.dot(X.T, y))
z = np.dot(X, b)
print y.T
print z.T

We don't even need to concern with the potential problems pseudo-inverse introduces at this moment. We can easily notice that $z$ is almost the same as $y$. When the difference between $y$ and $z$ occasionally occurs, the difference is quite negligible. 
 A: Universal approximation is not a theorem about matching output of the model to given sampled variables (your $n$), but to a given function. As such, you do need to consider situations where $n/p \rightarrow \infty$. A (sufficiently complex 2+ layer) neural network can fit any such curve
What you have done by looking at $n/p \rightarrow 0$ is shown that in high enough dimensional space, you can fit a hyperplane through any relatively small ($n << p$) number of points. It is not the same thing. However, it does have implications for over-fitting - the analogy is if you were to expand the number of features used in linear regression set by collecting lots of noisy data, then eventually you would find some probably spurious linear correlations and be able to fit a linear regression. 
The difference with neural networks is that a NN will fit an over-complex curve to simple data, but in order to perform your trick with linear regression, you need to add many more features than samples. It is not really a form of universal approximation, and has more in common with spurious correlations.
