I have a very basic understanding of Gaussian Processes. From what I understand, a Gaussian process for a set $X$, is the assignment of a Gaussian distribution to every element of the set. This is meant to expand the idea of a function to the case where we don't have total information about a function.

We can define the entropy of a probability distribution $p(x)$ as follows:

$$S = \int_X p(x) \log p(x) $$

with $x \in X$. How do we compute the entropy of a Gaussian Process?


This is just a guess, but I do not think that there can be a unique value for the entropy of a Gaussian Process.

The reason for this guess is that a GP is defined as a collection of random variables $X_{t_i}$ indexed on some set $T$, such that, for every finite sequence of indices $(t_1, ..., t_m) \in T^m$ you pick, the joint distribution of $(X_{t_1}, ..., X_{t_m})$ is a multivariate normal distribution.

This suggests that, if the sequence of indices change, the distribution parameters change, and it is my guess that the entropy will as well.

However, if you fix the indices (i.e. focus only on a set of points $(t_1, ..., t_m)$), then the entropy can be calculated, as the distribution $(X_{t_1}, ..., X_{t_m}) \sim \mathcal N(\mu, \Sigma_{t_1, ..., t_m})$, and the entropy of this multivariate normal is:

$$ H(t_1, ..., t_m) = \frac{1}{2}\log|\Sigma_{t_1, ..., t_m}| + \frac{m}{2}\log (2 \pi e) $$

  • $\begingroup$ I'm confused. I thought that GP was used for regression fit. If that's the case then the GP isn't constrained to be a probability distribution and so the entropy for a function that isn't a probability distribution isn't defined $\endgroup$ – Mellow Jan 19 '19 at 21:19
  • 1
    $\begingroup$ I don't follow. The GP isn't a probability distribution, it's a stochastic process. $\endgroup$ – InfProbSciX Jan 19 '19 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.