What is the entropy of a Gaussian Process? I have a very basic understanding of Gaussian Processes.  From what I understand, a Gaussian process for a set $X$, is the assignment of a Gaussian distribution to every element of the set.  This is meant to expand the idea of a function to the case where we don't have total information about a function.
We can define the entropy of a probability distribution $p(x)$ as follows:
$$S = \int_X p(x) \log p(x) $$
with $x \in X$.  How do we compute the entropy of a Gaussian Process?
 A: This is just a guess, but I do not think that there can be a unique value for the entropy of a Gaussian Process.
The reason for this guess is that a GP is defined as a collection of random variables $X_{t_i}$ indexed on some set $T$, such that, for every finite sequence of indices $(t_1, ..., t_m) \in T^m$ you pick, the joint distribution of $(X_{t_1}, ..., X_{t_m})$ is a multivariate normal distribution.
This suggests that, if the sequence of indices change, the distribution parameters change, and it is my guess that the entropy will as well.
However, if you fix the indices (i.e. focus only on a set of points $(t_1, ..., t_m)$), then the entropy can be calculated, as the distribution $(X_{t_1}, ..., X_{t_m}) \sim \mathcal N(\mu, \Sigma_{t_1, ..., t_m})$, and the entropy of this multivariate normal is:
$$ H(t_1, ..., t_m) = \frac{1}{2}\log|\Sigma_{t_1, ..., t_m}| + \frac{m}{2}\log (2 \pi e) $$
A: For stochastic processes, the usual generalisation of entropy is the entropy rate.
See here: https://jsri.srtc.ac.ir/article-1-24-en.pdf
[in case the link is broken in the future, look up "The Rate of Entropy for Gaussian Processes" by Leila Golshan]
