Can a joint second moment exist if the joint distribution cannot be derived?

Question

I am struggling with writing down a relationship between two variables and I am happy for each and every comment that points out, how I can overcome this problem:

Let $x\sim N(\mu,\Sigma), x \in\mathbb{R}^{N-1}$ be a vector that is multivariate normally distributed with mean parameter $\mu$ and covariance matrix $\Sigma$. $x_N$ is of such that $\sum_{i=1}^{N}x_i=1$. (Therefore: conditional on $x$ the scalar $x_N$ is deterministic defined as $1-\sum x$.) It is a standard result that $X_n$ (unconditional of $x$) as the sum of normal distributed variables is also normally distributed with mean parameter $1-\iota'x$ (where $\iota$ is a vector of ones) and variance $\iota'\Sigma\iota$. However, it is an obstacle for me to write down the joint distribution of $z:=(x,x_N)$ (which is apparently not normal due to the deterministic behavior of $x_N$ given $x$). However, I can easily sample from the joint distribution and it is not surprising to see for me that the expected value of $z=E(z)=(\mu,1-\iota'\mu)$. Sampling also gives me that the covariance between $x_N$ and $x_i$ ($i\in\{1,\ldots,N-1\}$) is what I would expect: $$cov(x_N,x_i)=cov(1-\sum_{j=1}^{N-1}x_j,x_i)=-\sum_{j=1}^{N-1}cov(x_j,x_i)$$ which is nothing but minus the sum of the $i$-th row of $\Sigma$. Can anyone help me to explain how to mathematically show that this is correct although I don't have the joint distribution of $z$?

Answer 1

Suppose that random variables $X_1,X_2,\ldots,X_{N-1}$ have a a multivariate normal distribution (jointly normal) with mean vector $\mathbf m = (\mu_1,\mu_2, \ldots, \mu_{N-1})$ and covariance matrix $C$. Define $$X_N = 1 - \sum_{i=1}^{N-1}X_i.$$ Then, $X_N$ is also normally distributed with mean $\displaystyle E[X_N] = \mu_N = 1 - \sum_{i=1}^{N-1} \mu_i$ and variance $$\sigma_N^2 = \operatorname{cov}(X_N,X_N) = \operatorname{cov}\left(\sum_{i=1}^{N-1}X_i, \sum_{j=1}^{N-1}X_j\right) = \sum_{i=1}^{N-1}\sum_{j=1}^{N-1} C_{i,j}.$$ Furthermore, the transformation $(X_1,X_2,\ldots, X_{N-1})\to (X_1,X_2,\ldots, X_{N-1}, X_N)$ is a linear transformation (well, actually an affine transformation if you want to nitpick) of jointly normal random variables, and so $(X_1,X_2,\ldots, X_{N-1}, X_N)$ also enjoys a jointly normal distribution with mean vector $\mathbf m^\prime = (\mathbf m, \mu_N) = (\mu_1,\mu_2, \ldots, \mu_{N-1}, \mu_N)$ and covariance matrix $\Sigma$ which is a bordered $N\times N$ matrix in which

• $C$ is the $(N-1)\times(N-1)$ submatrix in the upper left hand corner,

• $\Sigma_{N,N} = \sigma_N^2$ appears in the lower right hand corner

• the $(N,j)$-th entry in the bottom row of $\Sigma$, which is $\operatorname{cov}(X_N,X_j)$, has value $$\Sigma_{N,j} = \operatorname{cov}(X_N,X_j) = \operatorname{cov}\left(1-\sum_{i=1}^{N-1}X_i,X_j\right) = -\left(\sum_{i=1}^{N-1}\operatorname{cov}(X_i,X_j)\right) = -\sum_{i=1}^{N-1} C_{i,j},$$ that is, $\Sigma_{N,j}$ is the negative of the sum of the entries in the $j$-th column of $C$. Put another way, the column sums of $\Sigma$ equal $0$, and of you think about it a little, so do the row sums of $\Sigma$ equal $0$.

Note that all that we have used is the fact that covariance is a bilinear function.