What is the entropy of a composition of Gaussian Processes?

Question

I have a very basic understanding of Gaussian Processes. From what I understand, a Guassian process for a set $X$, is the assignment of a Gaussian distribution to every element of the set. This is meant to expand the idea of a function to the case where we don't have total information about a function.

I have a question about how we compute the entropy of a Gaussian Process over here. I also asked about whether or not we can compose Gaussian Processes and that question is here.

This question combines those other ones. Suppose we have entropies, $S_f$ and $S_g$, for two Gaussian Processes, $\mathcal{G}_f, \mathcal{G}_g$. Next, suppose we can compose two Gaussian Processes, $\mathcal{G}_f \cdot \mathcal{G}_g$. Is the entropy for the composition a simple function of the individual entropies?

ie $$S_{\mathcal{G}_f \cdot \mathcal{G}_g} = F(S_f, S_g)$$

Answer 1

Based on my other two answers to your questions, I do not think that such an expression exists because the entropy of the composition $f \circ g$ will explicitly depend on values of the function $g$, which is random.

One way to estimate the entropy for a fixed set of indices $t_1, ..., t_m$ value is by simulating the $\log(p_{f \circ g}(t))$.

Note that:

$$ p_G(g) = |2 \pi \Sigma_G(t_1, ..., t_m)|^{-1/2}exp(-0.5*g^T \Sigma_G^{-1}(t_1, ..., t_m)g)$$ $$ p_{f \circ g | g}(f) = |2 \pi \Sigma_F(g_1, ..., g_m)|^{-1/2}exp(-0.5*f^T \Sigma_F^{-1}(g_1, ..., g_m)f)$$

... and so, the joint density can be obtained by multiplying these. If we simulate a bunch of $g$s and $f$s and plug them into the log joint density and obtain a sample mean, we can obtain an estimate for the entropy.

Side note: 'training' a GP usually means working out the posterior distribution given a set of observed values. This posterior has a different covariance matrix (see this link) than a GP prior, hence a different entropy.