Maximizing likelihood for noise-free Gaussian process regression - Cross Validated most recent 30 from stats.stackexchange.com 2019-08-23T18:22:26Z https://stats.stackexchange.com/feeds/question/346551 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/346551 1 Maximizing likelihood for noise-free Gaussian process regression Luke https://stats.stackexchange.com/users/208544 2018-05-16T14:25:31Z 2018-05-16T14:25:31Z <p>In "Machine Learning: A Probabilistic Perspective" the maximum marginal likelihood optimization for the kernel hyperparameters is explained for the noisy observation case. I am dealing with a noise-free problem and want to derive the method for this case. If I understand correctly I could just set the varianace of the noise to zero ($\sigma_y^2=0$) basically substituting $K_y$ for $K$ in equations 15.22,15.23,15.24. Further if I understand equation 15.21 correctly the marginalization of $\boldsymbol{f}$ (Murphy calls it the latent Gaussian vector), gets rid of the real function values which seems counterproductive for the noise-free case.</p> <p>Basically, I came up with the following idea to derive the method for the noise-free case. By definition of the Gaussian process for a finite set of observations $\boldsymbol{f}$ an N-dimensional multivariate Gaussian distribution is given \begin{gather*} p(\boldsymbol{f}|\boldsymbol{X},\theta) \sim \mathcal{N}(\boldsymbol{f}|0,\boldsymbol{K}(\theta)) \end{gather*} where $\boldsymbol{X}$ are the input values and $\theta$ are the hyperparameters of the kernel. The logarithm is applied so that the gradient methods are more numerically stable and since it is monotonic it preserves extrema. \begin{align*} &amp; \log p(\boldsymbol{f}|\boldsymbol{X},\theta) =\log\mathcal{N}(\boldsymbol{f}|0,\boldsymbol{K}(\theta))\\ &amp;=-\frac{1}{2}\boldsymbol{f}^T\boldsymbol{K}(\theta)^{-1}\boldsymbol{f}-\frac{1}{2}\log |\boldsymbol{K}(\theta)|-\frac{N}{2}\log(2\pi) \end{align*}</p> <p>To maximize the log likelihood we use the derivative with respect to the hyperparameters $\theta$: \begin{align*} \frac{\partial}{\partial\theta_j} \log p(\boldsymbol{f}|\boldsymbol{X},\theta) &amp;= \frac{1}{2} \boldsymbol{f}^T \boldsymbol{K}(\theta)^{-1} \frac{\partial\boldsymbol{K}(\theta)}{\partial\theta_j} \boldsymbol{K}(\theta)^{-1} \boldsymbol{f}-\frac{1}{2}tr(\boldsymbol{K}(\theta)^{-1}\frac{\partial\boldsymbol{K}(\theta)}{\partial\theta_j}) \\ &amp;= \frac{1}{2}tr((\alpha\alpha^T-\boldsymbol{K}(\theta)^{-1})\frac{\partial \boldsymbol{K}(\theta)}{\partial \theta_j}) \\ \alpha &amp;= \boldsymbol{K}(\theta)^{-1}\boldsymbol{f} \end{align*}</p> <p>So I am not maximizing the log marginal likelihood but just the log likelihood. Is that a correct approach? I assume a library like scikit-learn would just set the noise variance to 0.</p>