# Maximum Prediction in Gaussian Process

A Gaussian process (GP) is defined as a collection of random variables with a joint Gaussian distribution (Rasmussen 2006). It is well known that given observations $\left \{ \mathbf{x},\mathbf{y}\right\}$ and given a covariance function such as the squared exponential $$k(x,x^{'})= \sigma exp(-(x-x^{'})^{2}/l^{2})$$ then for any new input point $x^{*}$ , the response $y^{*}$ can be calculated based on the multivaraite normal theory as follows $$\\\begin{pmatrix} \mathbf{y}\\y^{*} \end{pmatrix} \sim N(\mathbf{0},\begin{bmatrix} K & K_{*}^{T} \\ K_{*} & K_{**} \end{bmatrix})\\y^{*}|\mathbf{y} \sim N(K_{*}K^{-1}\mathbf{y},K_{**}-K_{*}K^{-1}K_{*}^{T})$$ I want to find the input points relating to the maximum and minimum prediction response over a domain of interest $x\in D$ , for instance in the figure below I want to find the input points $x^{*}=a,x^{*}=b$

What are the best and most efficient techniques that can be utilizaed for the gaussain process. Specifically, I am looking for possibly utilizing derivative information to provide a fast algortihm to find the inducing points $a,b$. I would also appreciate help in cases were $x^{*} \in \mathbb{R}^{d}$. Some good references on this topic would be of great help.

PS: The intent here is to estimate the input points which would cause a decrease or increase in the response

• Very, very, very closely related: stats.stackexchange.com/questions/193306/… – Sycorax Aug 10 '16 at 2:09
• Are you able to go back in time and record historical values? Because aside that you just fit your model (GPR or anything else) and simply look at the maximum and minimum values it attains in your domain of interest. Otherwise you are having a question about optimal design really and not about GPs. – usεr11852 Aug 10 '16 at 7:32
• @usεr11852 I cannot collect mnore historical data, I am searching for an efficient optimization algorithm which helps me find a and b especially in a high dimensional space. A full searcjh algorithm is not an option here – Wis Aug 11 '16 at 21:42
• @raK1: So you just want to fit "something" and then read out the extrema values of that estimated fit right? How large is your $d$ by the way? Usually cubic splines for examples are fairly fast to evaluate and can handle low ($d \leq 3$) fairly well. Otherwise I see not real question, you have decided on your technique (GPs), just fit it the model and move forward with it. You don't say what you have tried yet so it is difficult to contextualise the issue you experience. – usεr11852 Aug 11 '16 at 22:18
• @raK1: Fitting a Gaussian process and finding it's extrema are two different tasks. In the first task a simple BFGS will suffice at first instance. Don't complicate things unnecessarily. First try it, see if the standard procedure is good enough and then expand on it (you still haven't said how big $d$ is by the way). In the second task you need to evaluate a model multiple time and it is effectively a programming task rather than a statistics task. – usεr11852 Aug 12 '16 at 8:20