I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. [here][1] and [here][2].) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related. 

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured. 

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values. 

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data? 

Edit: So there is [literature][3] showing that neural nets predictors can be viewed as GPs (in the infinite width limit). If we say the prediction function is a GP, what does that tell us about the loss? The loss is in some sense a transformation of $n$ points of the prediction function. 


  [1]: https://arxiv.org/pdf/1405.4604.pdf
  [2]: http://proceedings.mlr.press/v38/choromanska15.pdf
  [3]: https://arxiv.org/pdf/1711.00165.pdf