Viewing a neural net loss function as a Gaussian Process

Question

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here$^1$ and here$^2$.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?

Edit: So there is literature showing that neural nets predictors can be viewed as GPs (in the infinite width limit). If we say the prediction function is a GP, what does that tell us about the loss? The loss is in some sense a transformation of $n$ points of the prediction function.

---

$^1$ Pascanu, Razvan, et al. "On the saddle point problem for non-convex optimization." arXiv preprint arXiv:1405.4604 (2014). https://arxiv.org/pdf/1405.4604.pdf

$^2$ Choromanska, Anna, et al. "The loss surfaces of multilayer networks." Artificial intelligence and statistics. PMLR, 2015. http://proceedings.mlr.press/v38/choromanska15.pdf

Answer 1

Disclaimer: I just glimpsed at the linked articles. My answer focuses solely on Gaussian processes per se.

Your example

If you understand your example $f(x)=x^2+Y$ as a random function of $x$, then it is a Gaussian process, if and only if $Y$ is a Gaussian random variable. If it has any other distribution, it is still a stochastic process but not a Gaussian one. But no matter what kind of process, you can determine the mean and covariance function. The mean function is $m(x) = x^2 + \mathbb{E}[Y]$ and the covariance function $C$ is $$C(x_1,x_2)=\text{Cov}(f(x_1), f(x_2))= \text{Cov}(x_1^2 + Y, x_2^2 + Y)=\text{Cov}(Y,Y)=\text{Var}[Y].$$ This is because $x_1$ and $x_2$ are non-random, which means the respective covariance terms are zero.

This particular covariance function does not depend on $x$ i.e. it is constant. This reflects the fact that your set of random functions is the parabola $x^2$ which is just randomly shifted up and down as a whole by the values of $Y.$

Parametric families

If you can express your parametric family as a span of basis functions, as is the case for example for polynomials, you can easily turn them into a Gaussian process. Say your family consists of functions of the form $f(x)=\sum \alpha_i \phi_i(x)$ for a set of some basis functions $\phi_i$ then you can turn those into Gaussian random functions simply by turning the coefficients $\alpha_i$ into Gaussian random variables $Y_i$ to arrive at the Gaussian process $$ G(x) = \sum_i \phi_i(x) Y_i.$$

This works only because linear combinations of Gaussian variables are again Gaussian variables, and explains what is so special about the "Gaussian" in Gaussian processes. For other distributions, including discrete ones, this is no longer true.

The idea is straightforward for finite dimensional spaces, but with the proper technical assumptions even possible for infinite dimensional spaces.

Final remark

If you want to discuss how functions look or what they do "on average" you need to find a way to express probabilities for functions. As demonstrated above, Gaussian process provide an easy and flexible way to turn any space of functions into a space of random functions. The source and purpose of those functions, i.e. whether they are regression functions or loss functions or time series of beaver dams built, does not matter the least.

Answer 2

Neural Networks as Gaussian Processes

Consider a neural network with only one layer (i.e. no hidden layers, i.e. logistic regression): $$\operatorname{reg}: \mathbb{R}^N \to \mathbb{R}^M : \boldsymbol{x} \mapsto \boldsymbol{s} = \boldsymbol{W} \boldsymbol{x}.$$ If we replace the entries in $\boldsymbol{W} \in \mathbb{R}^{M \times N}$ by random values, such that $w_{ij} \sim \mathcal{N}(0, \sigma_w^2)$, the resulting function will be a random/stochastic process.

Now, let $\boldsymbol{w}_i$ be a row of $\boldsymbol{W}$, such that $$s_i = \boldsymbol{w}_i \boldsymbol{x} = \sum_{j=1}^N w_{ij} x_j,$$ we can use the central limit theorem to conclude that $s_i$ follows a Gaussian distribution if $N \to \infty$. Therefore, a large number of inputs ($N$) turns the random process into a Gaussian process (because the outputs are now Gaussian).

This is exactly the idea presented in your last piece of literature (Lee, 2018). Although Lee et al. write about infinite width in every layer, I would argue that you only really need it in the penultimate layer (i.e. the inputs to the final layer). Having infinite width everywhere just makes the computation of the mean and covariance functions tractable (at least for ReLU networks).

The Effect of Loss Functions

A loss function by itself will never be a Gaussian process because there is typically no randomness in a loss function. This being said, the combination of neural network and loss function can give rise to a random process. In order to assess whether this random process will still be Gaussian depends on the loss function itself. I believe that there are no practical loss functions that would preserve Gaussianity. E.g. when using the mean squared error, $(\operatorname{reg}(\boldsymbol{x} \mathbin{;} \boldsymbol{w}) - y)^2,$ it should be clear that the loss values will not be Gaussian.

After skimming the papers that are referenced in your question, I am not entirely sure whether they really talk about loss functions as Gaussian processes:

• Pascanu et al. (2014) mention that they use random loss functions, sampled from a Gaussian process. This would be using GPs exactly as how you described them: a distribution of functions.
• Choromanska et al. (2015) seem to try to prove that a ReLU network with some loss function that uses randomness is related to a Gaussian process. At least that would be my interpretation since I do not know much about spin-glass models.