In "Machine Learning: A Probabilistic Perspective" the maximum marginal likelihood optimization for the kernel hyperparameters is explained for the noisy observation case. I am dealing with a noise-free problem and want to derive the method for this case. If I understand correctly I could just set the varianace of the noise to zero ($\sigma_y^2=0$) basically substituting $K_y$ for $K$ in equations 15.22,15.23,15.24. Further if I understand equation 15.21 correctly the marginalization of $\boldsymbol{f}$ (Murphy calls it the latent Gaussian vector), gets rid of the real function values which seems counterproductive for the noise-free case.

Basically, I came up with the following idea to derive the method for the noise-free case. By definition of the Gaussian process for a finite set of observations $\boldsymbol{f}$ an N-dimensional multivariate Gaussian distribution is given \begin{gather*} p(\boldsymbol{f}|\boldsymbol{X},\theta) \sim \mathcal{N}(\boldsymbol{f}|0,\boldsymbol{K}(\theta)) \end{gather*} where $\boldsymbol{X}$ are the input values and $\theta$ are the hyperparameters of the kernel. The logarithm is applied so that the gradient methods are more numerically stable and since it is monotonic it preserves extrema. \begin{align*} & \log p(\boldsymbol{f}|\boldsymbol{X},\theta) =\log\mathcal{N}(\boldsymbol{f}|0,\boldsymbol{K}(\theta))\\ &=-\frac{1}{2}\boldsymbol{f}^T\boldsymbol{K}(\theta)^{-1}\boldsymbol{f}-\frac{1}{2}\log |\boldsymbol{K}(\theta)|-\frac{N}{2}\log(2\pi) \end{align*}

To maximize the log likelihood we use the derivative with respect to the hyperparameters $\theta$: \begin{align*} \frac{\partial}{\partial\theta_j} \log p(\boldsymbol{f}|\boldsymbol{X},\theta) &= \frac{1}{2} \boldsymbol{f}^T \boldsymbol{K}(\theta)^{-1} \frac{\partial\boldsymbol{K}(\theta)}{\partial\theta_j} \boldsymbol{K}(\theta)^{-1} \boldsymbol{f}-\frac{1}{2}tr(\boldsymbol{K}(\theta)^{-1}\frac{\partial\boldsymbol{K}(\theta)}{\partial\theta_j}) \\ &= \frac{1}{2}tr((\alpha\alpha^T-\boldsymbol{K}(\theta)^{-1})\frac{\partial \boldsymbol{K}(\theta)}{\partial \theta_j}) \\ \alpha &= \boldsymbol{K}(\theta)^{-1}\boldsymbol{f} \end{align*}

So I am not maximizing the log marginal likelihood but just the log likelihood. Is that a correct approach? I assume a library like scikit-learn would just set the noise variance to 0.

  • $\begingroup$ is it fair to say that you're basically asking why people typically choose $\theta$ by maximum marginal likelihood rather than just maximizing the regular likelihood? $\endgroup$ – jld May 16 '18 at 15:09
  • $\begingroup$ @Chaconne: Yes, kind of. Also: Can I just maximize the likelihood in the noise-free case instead and is my approach correct? $\endgroup$ – Luke May 17 '18 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.