I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here$^1$ and here$^2$.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.
From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.
More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.
Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?
Edit: So there is literature showing that neural nets predictors can be viewed as GPs (in the infinite width limit). If we say the prediction function is a GP, what does that tell us about the loss? The loss is in some sense a transformation of $n$ points of the prediction function.
$^1$ Pascanu, Razvan, et al. "On the saddle point problem for non-convex optimization." arXiv preprint arXiv:1405.4604 (2014). https://arxiv.org/pdf/1405.4604.pdf
$^2$ Choromanska, Anna, et al. "The loss surfaces of multilayer networks." Artificial intelligence and statistics. PMLR, 2015. http://proceedings.mlr.press/v38/choromanska15.pdf