How are infinite neural networks implemented in practice? Consider for example Neural Tangents. They claim they allow to

define, train, and evaluate infinite networks as easily as finite ones.

If their width is infinite (and thus they have an infinite number of parameters) how exactly are these neural networks or layers represented and connected to other layers in practice?

From what I read, these networks are equivalent to Gaussian Processes. To my knowledge, GPs are fully defined by their covariance matrix or function (i.e. a Kernel describing how two inputs covary), but GPs don't have an infinite number of parameters per se. Sure GPs are non-parameteric in that their ability to interpolate data grows with the data, but Kernels still have parameters governing e.g. the "range" of interaction in the covariance matrix, e.g. how smooth the process can be. 
Simple example exploring the relationship with GPs
Let's say we use a GP in 1D as an example. In a GP, the input could just be one variable (e.g. a single real value $x$), so if we feed it to an "infinitely-wide neural network", how exactly is that equivalent to a layer of infinite width? E.g. would an infinitely wide layer simply work as GP kernel $K(x,x')$ that takes (in 1D) a $\mathbf{x}$ vector as its input and it outputs a variable $\mathbf{y}$ of the same size as $\mathbf{x}$ and distributed as a GP? If so, wouldn't that be a width of 1? (one input $\rightarrow$ one output)
 A: *

*A neural network
$
NN(x) = \sum_{j=1}^M w_j \sigma(x\cdot b_j)
$


*If we can apply the central limit theorem then
$NN(x) \sim \mathcal{N}(0, K)$
as $M$ tends to infinity.


*Assume that $w$ and $b$ are iid with zero mean and variance $s_w, s_b$. Then
$E_w[NN(x)] = 0$ and $V[NN(x)] = E[NN(x) NN(x)^T] = s^2_w ME_b[\sigma(x\cdot b) \sigma(x\cdot b)^T]$. Make this finite by letting $s^2_w$ scale with $M$. Thus the central limit theorem can be applied.


*The infinite width comes from the CLT: If the width is infinite and the assumptions from 3 hold then the output of an infinitely wide NN is simply a normally distributed variable. The expected value is zero so all we need is the covariance matrix $K$.
A: Given your data set $(X,Y)$, what you do with a neural network is actually trying to find a (very complex) function $f$ that explains the data: $f(X)=Y + \epsilon$, where $\epsilon$ is the residual.
Instead of creating one such highly dimensional complex function with a neural net, the idea of Gaussian Processes is to model a distribution over all functions that explain the data. This is of course non-parametric in a sense that there are infinitely many such functions. Theoretically, this is equivalent to a infinite-dimensional neural net that is able to model such any function. 
