Return to Question

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here and here.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?

Edit: So there is literature showing that neural nets predictors can be viewed as GPs (in the infinite width limit). If we say the prediction function is a GP, what does that tell us about the loss? The loss is in some sense a transformation of $n$ points of the prediction function.

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here$^1$ and here$^2$.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?

Edit: So there is literature showing that neural nets predictors can be viewed as GPs (in the infinite width limit). If we say the prediction function is a GP, what does that tell us about the loss? The loss is in some sense a transformation of $n$ points of the prediction function.

$^1$ Pascanu, Razvan, et al. "On the saddle point problem for non-convex optimization." arXiv preprint arXiv:1405.4604 (2014). https://arxiv.org/pdf/1405.4604.pdf

$^2$ Choromanska, Anna, et al. "The loss surfaces of multilayer networks." Artificial intelligence and statistics. PMLR, 2015. http://proceedings.mlr.press/v38/choromanska15.pdf

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here and here.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here and here.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?

Edit: So there is literature showing that neural nets predictors can be viewed as GPs (in the infinite width limit). If we say the prediction function is a GP, what does that tell us about the loss? The loss is in some sense a transformation of $n$ points of the prediction function.

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here and here.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? And if not, what's the best way to view it? What's the justification of regarding neural net loss functions as GPs with respect to the input data?

I know that GPs are usually used for regression and function-estimation tasks. But I've seen some literature in which neural net error functions are viewed as GPs, and properties of those error functions are then deduced (e.g. here and here.) For example, the first paper applies results about critical points of Gaussian random fields -- namely that the error and index of a critical point are inversely related.

From my understanding, a GP is a distribution over functions, where there is some "mean function" and a covariance kernel $k(x,x')$ that describes the covariance between the values $f(x)$ and $f(x')$. However I don't see how we can describe a neural net loss function (or any function that has a parametric form) as a GP, which seems more unstructured.

More specifically, I'm wondering whether we can view a parametric random function as a Gaussian process. For example, $f(x,Y) = x^2 + Y$ where $Y$ is a random variable with some known distribution. Here $f$ is very specific and only takes on a certain set of functions. And if $Y$ is a discrete r.v., then $f(x,Y)$ only takes on discrete values.

Can we view this kind of setup as a GP? If so, how can the covariance matrix be derived? I know it wouldn't be one of the typical ones used for regression. And if not, what's the best way to view a loss function as a random function? What's the justification of regarding neural net loss functions as GPs with respect to the input data?