# How are infinite neural networks implemented in practice?

Consider for example Neural Tangents. They claim they allow to

define, train, and evaluate infinite networks as easily as finite ones.

If their width is infinite (and thus they have an infinite number of parameters) how exactly are these neural networks or layers represented and connected to other layers in practice?

From what I read, these networks are equivalent to Gaussian Processes. To my knowledge, GPs are fully defined by their covariance matrix or function (i.e. a Kernel describing how two inputs covary), but GPs don't have an infinite number of parameters per se. Sure GPs are non-parameteric in that their ability to interpolate data grows with the data, but Kernels still have parameters governing e.g. the "range" of interaction in the covariance matrix, e.g. how smooth the process can be.

### Simple example exploring the relationship with GPs

Let's say we use a GP in 1D as an example. In a GP, the input could just be one variable (e.g. a single real value $$x$$), so if we feed it to an "infinitely-wide neural network", how exactly is that equivalent to a layer of infinite width? E.g. would an infinitely wide layer simply work as GP kernel $$K(x,x')$$ that takes (in 1D) a $$\mathbf{x}$$ vector as its input and it outputs a variable $$\mathbf{y}$$ of the same size as $$\mathbf{x}$$ and distributed as a GP? If so, wouldn't that be a width of 1? (one input $$\rightarrow$$ one output)

• It appears that you're asking how to represent and train infinite neural networks, and you link to a code repository which purports to represent and train infinite neural networks. What have you learned by reviewing the resources listed in the repository? What do you understand and what parts of it need more explanation?
– Sycorax
Commented Jun 1, 2020 at 14:50
• Thanks @SycoraxsaysReinstateMonica I updated the OP to try to provide more info about what I understand so far.
– Josh
Commented Jun 1, 2020 at 15:29

1. A neural network $$NN(x) = \sum_{j=1}^M w_j \sigma(x\cdot b_j)$$

2. If we can apply the central limit theorem then $$NN(x) \sim \mathcal{N}(0, K)$$ as $$M$$ tends to infinity.

3. Assume that $$w$$ and $$b$$ are iid with zero mean and variance $$s_w, s_b$$. Then $$E_w[NN(x)] = 0$$ and $$V[NN(x)] = E[NN(x) NN(x)^T] = s^2_w ME_b[\sigma(x\cdot b) \sigma(x\cdot b)^T]$$. Make this finite by letting $$s^2_w$$ scale with $$M$$. Thus the central limit theorem can be applied.

4. The infinite width comes from the CLT: If the width is infinite and the assumptions from 3 hold then the output of an infinitely wide NN is simply a normally distributed variable. The expected value is zero so all we need is the covariance matrix $$K$$.

• Thanks, what is $M$ supposed to represent? (i.e. why the sum? is that a weighted averaging layer?)
– Josh
Commented Jun 1, 2020 at 17:31

Given your data set $$(X,Y)$$, what you do with a neural network is actually trying to find a (very complex) function $$f$$ that explains the data: $$f(X)=Y + \epsilon$$, where $$\epsilon$$ is the residual.

Instead of creating one such highly dimensional complex function with a neural net, the idea of Gaussian Processes is to model a distribution over all functions that explain the data. This is of course non-parametric in a sense that there are infinitely many such functions. Theoretically, this is equivalent to a infinite-dimensional neural net that is able to model such any function.

• not related to the discussed issue Commented Jun 2, 2020 at 15:46