I have a very basic understanding of Gaussian Processes. From what I understand, a Guassian process for a set $X$, is the assignment of a Gaussian distribution to every element of the set. This is meant to expand the idea of a function to the case where we don't have total information about a function.

I have a question about how we compute the entropy of a Gaussian Process over here. I also asked about whether or not we can compose Gaussian Processes and that question is here.

This question combines those other ones. Suppose we have entropies, $S_f$ and $S_g$, for two Gaussian Processes, $\mathcal{G}_f, \mathcal{G}_g$. Next, suppose we can compose two Gaussian Processes, $\mathcal{G}_f \cdot \mathcal{G}_g$. Is the entropy for the composition a simple function of the individual entropies?

ie $$S_{\mathcal{G}_f \cdot \mathcal{G}_g} = F(S_f, S_g)$$


1 Answer 1


Based on my other two answers to your questions, I do not think that such an expression exists because the entropy of the composition $f \circ g$ will explicitly depend on values of the function $g$, which is random.

One way to estimate the entropy for a fixed set of indices $t_1, ..., t_m$ value is by simulating the $\log(p_{f \circ g}(t))$.

Note that:

$$ p_G(g) = |2 \pi \Sigma_G(t_1, ..., t_m)|^{-1/2}exp(-0.5*g^T \Sigma_G^{-1}(t_1, ..., t_m)g)$$ $$ p_{f \circ g | g}(f) = |2 \pi \Sigma_F(g_1, ..., g_m)|^{-1/2}exp(-0.5*f^T \Sigma_F^{-1}(g_1, ..., g_m)f)$$

... and so, the joint density can be obtained by multiplying these. If we simulate a bunch of $g$s and $f$s and plug them into the log joint density and obtain a sample mean, we can obtain an estimate for the entropy.

Side note: 'training' a GP usually means working out the posterior distribution given a set of observed values. This posterior has a different covariance matrix (see this link) than a GP prior, hence a different entropy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.