Gaussian process likelihood function - Cross Validated most recent 30 from stats.stackexchange.com 2019-09-19T17:30:12Z https://stats.stackexchange.com/feeds/question/397132 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/397132 0 Gaussian process likelihood function mamun https://stats.stackexchange.com/users/77602 2019-03-12T18:25:58Z 2019-03-20T13:44:09Z <p>I'm trying to understand the likelihood function in Gaussian Process. The book by Rasmussen et al. defined Gaussian Process lml as </p> <p><span class="math-container">$$log~p(y|X) = -\frac{1}{2}y^T\alpha-\sum log L_{ii} - \frac{N}{2}log (2\pi)$$</span> </p> <p>Where <span class="math-container">$\alpha$</span> is computed from the lower triangular matrix of the cholesky decomposition (I'm omitting the noise for simplicity): <span class="math-container">$$L = cholesky(K)$$</span> <span class="math-container">$$\alpha = L^T\backslash L\backslash y$$</span></p> <p>The implementation in scikit learn is:</p> <pre><code>L = cholesky(K, lower=True) alpha = cho_solve((L, True), y_train) log_likelihood_dims = -0.5 * np.einsum("ik,ik-&gt;k", y_train, alpha) log_likelihood_dims -= np.log(np.diag(L)).sum() log_likelihood_dims -= K.shape / 2 * np.log(2 * np.pi) log_likelihood = log_likelihood_dims.sum(-1) </code></pre> <p>If we compare the <span class="math-container">$\alpha$</span> given by the equation above and sklearn implementation, they are not same. sklearn will give <span class="math-container">$\alpha = L^{-1}y$</span> while the equation says <span class="math-container">$\alpha=L^T/(L^{-1}y)$</span>. My hunch is both of them are equivalent expression as the numerical value of lml is not something meaningful, i.e., we are only interested in the hyperparameters that optimizes lml. So, instead of computing the true lml, we are computing a surrogate lml?</p> <p>On a similar note, I saw several different likelihood in the GPML code written in MATLAB. So, does that mean we can choose any suitable likelihood function (or cost function) to optimize hyperparameter depending on the problem description?</p> https://stats.stackexchange.com/questions/397132/-/398526#398526 0 Answer by Skullgreymon for Gaussian process likelihood function Skullgreymon https://stats.stackexchange.com/users/185695 2019-03-20T13:44:09Z 2019-03-20T13:44:09Z <p>Actually, this code is computing <span class="math-container">$\alpha=L^T\backslash L\backslash y$</span>. With <em>cho_solve</em> you are solving the original system taking advantage of the cholesky descomposition. </p> <p>You want to compute the likelihood (or log-likelihood) because it shows the goodness of the learned model. To compare the model and to learn the hyperparameters you want to evaluate the exact function.</p> <p>Using variational inference it is used the ELBO (evidence lower bound) as objective function. This is a lower bound of the log-likelihood because in this kind of models you are not able to compute the exact likelihood. </p>