I have a very basic understanding of Gaussian Processes. From what I understand, a Gaussian process for a set $X$, is the assignment of a Gaussian distribution to every element of the set. This is meant to expand the idea of a function to the case where we don't have total information about a function.

We can define the entropy of a probability distribution $p(x)$ as follows:

$$S = \int_X p(x) \log p(x) $$

with $x \in X$. How do we compute the entropy of a Gaussian Process?


2 Answers 2


For stochastic processes, the usual generalisation of entropy is the entropy rate.

See here: https://jsri.srtc.ac.ir/article-1-24-en.pdf

[in case the link is broken in the future, look up "The Rate of Entropy for Gaussian Processes" by Leila Golshan]


This is just a guess, but I do not think that there can be a unique value for the entropy of a Gaussian Process.

The reason for this guess is that a GP is defined as a collection of random variables $X_{t_i}$ indexed on some set $T$, such that, for every finite sequence of indices $(t_1, ..., t_m) \in T^m$ you pick, the joint distribution of $(X_{t_1}, ..., X_{t_m})$ is a multivariate normal distribution.

This suggests that, if the sequence of indices change, the distribution parameters change, and it is my guess that the entropy will as well.

However, if you fix the indices (i.e. focus only on a set of points $(t_1, ..., t_m)$), then the entropy can be calculated, as the distribution $(X_{t_1}, ..., X_{t_m}) \sim \mathcal N(\mu, \Sigma_{t_1, ..., t_m})$, and the entropy of this multivariate normal is:

$$ H(t_1, ..., t_m) = \frac{1}{2}\log|\Sigma_{t_1, ..., t_m}| + \frac{m}{2}\log (2 \pi e) $$

  • $\begingroup$ I'm confused. I thought that GP was used for regression fit. If that's the case then the GP isn't constrained to be a probability distribution and so the entropy for a function that isn't a probability distribution isn't defined $\endgroup$
    – piccolo
    Commented Jan 19, 2019 at 21:19
  • 2
    $\begingroup$ I don't follow. The GP isn't a probability distribution, it's a stochastic process. $\endgroup$
    – adityar
    Commented Jan 19, 2019 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.