Is the sample log marginal likelihood divided by its dimensionality n constant as n increases?

Question

Hi everyone and thank you all in advance!

I am a physicist working on Multivariate Gaussian distributions (not with a really strong theoretical math background). Let's have a sample drawn from a particular distribution with zero mean and correlations provided by a covariance matrix and random Gaussian independent noise. Its matrix elements are defined by a kernel function quantifying observation correlations among input points. This kernel is characterized by a fixed set of hyperparameters subject to optimization. I was working with the expression of log-marginal likelihood. Evaluating our data set over the given distribution yields,

$$ \log p \left(y \vert x \right) = - \frac{1}{2} y^{T} \left(K + \sigma_{n} \mathbb{I}\right)^{-1}y - \log \vert K + \sigma^{2}_{n} \mathbb{I} \vert - \frac{n}{2} \log 2 \pi $$

Where $ K + \sigma^{2}_{n} \mathbb{I} $ is the matrix whose elements are determined by the kernel function plus a diagonal contribution stemming from random Gaussian additive noise over the observations. The idea is interpreting the data set as a Gaussian Process so that more datapoints can be included extending the covariance matrix by evaluating proper elements of the kernel function among new and previous data points pair-wise.

If the observations $y$ are properly described by the Gaussian Distribution, as long as more observations are considered, I have the intuition that some sort of stationary has to be derived from the log p expression. In other words I feel that $\frac{\log p \left(y \vert x \right)}{dimensionality\left(y\right)} $ has to be constant as $ n = dimensionality\left(y\right)$ increases.

Term by term, I clearly see that dividing by $n$ the last two contributions are constant. Under the assumption that covariance matrix elements depend on distance and diminishes towards zero as domain point separation increases, the determinant term is equivalent to the n-power of the diagonal elements if n is sufficiently big. Thus taking logarithm of such expression makes it proportional to $n$.

Nevertheless I am not able to workout the first element. The one involving the observation vectors. I feel it should be related to the statistical properties of observations. For example, considering the dummy set of gaussian independent noise random variables, $K = \sigma^{2}_{n}\mathbb{I}$,

$$ \frac{1}{n} y^{T} \left(\sigma^{2}_{n} \mathbb{I}\right)^{-1}y = \frac{1}{n} y^{T} \left(\frac{1}{\sigma^{2}_{n}} \mathbb{I}\right)y = \frac{1}{n} \frac{1}{\sigma^{2}_{n}} y^{T} y = \frac{1}{\sigma^{2}_{n}} \frac{\sum_{i}y_{i}^{2}}{n} = \frac{1}{\sigma^{2}_{n}} \langle y^{2} \rangle $$

Is it already known to mathematicians a result for this matter? or rather this has to be worked around or even it is not a correct hypothesis.

Thank you all again.

Answer 1

You are correct.

In the special case you give, where the $y_i$ are independent, the law of large numbers says that $\frac{1}{\sigma^2_nn}\sum y_i^2$ converges (in probability or almost surely or in mean square) to its expected value, which is 1.

More generally, if $y\sim N(0, V_n)$ then $V_n^{-1/2}y\sim N(0,1)$ (assuming $V_n^{-1/2}$ exists), so $y^TV_n^{-1}y$ has the same distribution as if $V_n$ was the identity and $y\sim N(0,1)$. So, at least as long as the matrix $K+\sigma^2I_n$ stays bounded away from being singular, it will still be true that the quadratic form converges to a constant when scaled by $1/n$.