Entropy of a Gaussian Process: Log(Determinant(CovarianceMatrix)) I want to be able to compute the entropy of a Gaussian Process.  To that end, I have a simple example in GPFLow.  I have a latent function which I sample with two different levels of noise, L1 and L2 where Noise(L1)>Noise(L2).  I then regress the Gaussian process in a small range over my function and compute the covariance matrix, the determinant of this, and then the log of the determinant as the entropy.  I have found a few things.  Firstly, as I compute the covariance matrix for an increasingly dense set of points in a fixed interval, the determinant goes to zero (0.0, actually).  Secondly, as I decrease the noise from L1 to L2, the log of the determinant does not go toward zero.  Are my reasoning and computations correct?
Here some extra notes about the code I am using.
Here is the code I am using:
import gpflow
import numpy as np
N = 300
noiseSize = 0.01
X = np.random.rand(N,1)
Y = np.sin(12*X) + 0.66*np.cos(25*X)  + np.random.randn(N,1)*noiseSize + 3
k = gpflow.kernels.Matern52(1, lengthscales=0.3)
m = gpflow.models.GPR(X, Y, kern=k)
m.likelihood.variance = 0.01
aRange = np.linspace(0.1,0.9,200)
newRange = []
for point in aRange:
    newRange.append([point])
covMatrix = m.predict_f_full_cov(newRange)[1][0]
import math
print("Determinant: " + str(np.linalg.det(covMatrix)))
print(-10*math.log(np.linalg.det(covMatrix)))

 A: So, first things first, the entropy of a multivariate normal (and a GP, given a fixed set of points on which it's evaluated) only depends on its covariance matrix.
Answers to your questions:


*

*Yes - when you make the set $X$ more and more dense, you're making the covariance matrix larger and larger, and for many simple covariance kernels, this makes the determinant smaller and smaller. My guess is that this is due to the fact that determinants of large matrices have a lot of product terms (see the Leibniz formula) and products of terms less than one tend to zero faster than their sums. You can verify this easily:



Code for this:
import numpy as np
import matplotlib.pyplot as plt
import sklearn.gaussian_process.kernels as k

plt.style.use("ggplot"); plt.ion()

n = np.linspace(2, 25, 23, dtype = int)
d = np.zeros(len(n))

for i in range(len(n)):
    X = np.linspace(-1, 1, n[i]).reshape(-1, 1)
    S = k.RBF()(X)
    d[i] = np.log(np.linalg.det(S))

plt.scatter(n, d)
plt.ylabel("Log Determinant of Covariance Matrix")
plt.xlabel("Dimension of Covariance Matrix")

Before moving onto the next point, do note that the entropy of a multivariate normal also has a contribution from size of the matrix, so even though the determinant shoots off to zero, there's a small contribution from the dimension.


*With decreasing noise, as one would expect, the entropy & determinant do decrease but not tend to zero exactly; they'll decrease to the determinant due to the other kernels present in the covariance. For the demonstration below, the dimension of the covariance is kept constant ($10*10$) and the noise level is increased from 0:



Code:
e = np.logspace(1, -10, 30)
d = np.zeros(len(e))
X = np.linspace(-1, 1, 10).reshape(-1, 1)

for i in range(len(e)):
    S = (k.RBF() + k.WhiteKernel(e[i])) (X)
    d[i] = np.log(np.linalg.det(S))

e = np.log(e)

plt.scatter(e, d)
plt.ylabel("Log Determinant")
plt.xlabel("Log Error")

