I am struggling with writing down a relationship between two variables and I am happy for each and every comment that points out, how I can overcome this problem:

Let $x\sim N(\mu,\Sigma), x \in\mathbb{R}^{N-1}$ be a vector that is multivariate normally distributed with mean parameter $\mu$ and covariance matrix $\Sigma$. $x_N$ is of such that $\sum_{i=1}^{N}x_i=1$. (Therefore: conditional on $x$ the scalar $x_N$ is deterministic defined as $1-\sum x$.) It is a standard result that $X_n$ (unconditional of $x$) as the sum of normal distributed variables is also normally distributed with mean parameter $1-\iota'x$ (where $\iota$ is a vector of ones) and variance $\iota'\Sigma\iota$. However, it is an obstacle for me to write down the joint distribution of $z:=(x,x_N)$ (which is apparently not normal due to the deterministic behavior of $x_N$ given $x$). However, I can easily sample from the joint distribution and it is not surprising to see for me that the expected value of $z=E(z)=(\mu,1-\iota'\mu)$. Sampling also gives me that the covariance between $x_N$ and $x_i$ ($i\in\{1,\ldots,N-1\}$) is what I would expect: $$cov(x_N,x_i)=cov(1-\sum_{j=1}^{N-1}x_j,x_i)=-\sum_{j=1}^{N-1}cov(x_j,x_i)$$ which is nothing but minus the sum of the $i$-th row of $\Sigma$. Can anyone help me to explain how to mathematically show that this is correct although I don't have the joint distribution of $z$?

  • $\begingroup$ I believe you mean $N$ instead of $N-1$ in the first line. Perhaps many of your difficulties have been addressed at stats.stackexchange.com/questions/178389? $\endgroup$
    – whuber
    Commented Oct 25, 2015 at 19:27
  • $\begingroup$ Thank you @whuber! The $N-1$ is correct in my question, but in fact your linked question is related to mine: The support of the joint distribution $(x,x_N)$ falls into an $N-1$ dimensional subspace. However, I can not point out from the answer of @air how to come up with the covariance matrix. Can you give me a hint? $\endgroup$ Commented Oct 25, 2015 at 19:47
  • 1
    $\begingroup$ You don't need the joint distribution to determine the covariance of $X_N$ with any other $X_i$: use the law of the unconscious statistician to determine the expectation of $g(X_N,X_i)$ from the distribution of just the $X_i, i = 1,2,\ldots,N-1$. $\endgroup$ Commented Oct 25, 2015 at 20:30

1 Answer 1


Suppose that random variables $X_1,X_2,\ldots,X_{N-1}$ have a a multivariate normal distribution (jointly normal) with mean vector $\mathbf m = (\mu_1,\mu_2, \ldots, \mu_{N-1})$ and covariance matrix $C$. Define $$X_N = 1 - \sum_{i=1}^{N-1}X_i.$$ Then, $X_N$ is also normally distributed with mean $\displaystyle E[X_N] = \mu_N = 1 - \sum_{i=1}^{N-1} \mu_i$ and variance $$\sigma_N^2 = \operatorname{cov}(X_N,X_N) = \operatorname{cov}\left(\sum_{i=1}^{N-1}X_i, \sum_{j=1}^{N-1}X_j\right) = \sum_{i=1}^{N-1}\sum_{j=1}^{N-1} C_{i,j}.$$ Furthermore, the transformation $(X_1,X_2,\ldots, X_{N-1})\to (X_1,X_2,\ldots, X_{N-1}, X_N)$ is a linear transformation (well, actually an affine transformation if you want to nitpick) of jointly normal random variables, and so $(X_1,X_2,\ldots, X_{N-1}, X_N)$ also enjoys a jointly normal distribution with mean vector $\mathbf m^\prime = (\mathbf m, \mu_N) = (\mu_1,\mu_2, \ldots, \mu_{N-1}, \mu_N)$ and covariance matrix $\Sigma$ which is a bordered $N\times N$ matrix in which

  • $C$ is the $(N-1)\times(N-1)$ submatrix in the upper left hand corner,

  • $\Sigma_{N,N} = \sigma_N^2$ appears in the lower right hand corner

  • the $(N,j)$-th entry in the bottom row of $\Sigma$, which is $\operatorname{cov}(X_N,X_j)$, has value $$\Sigma_{N,j} = \operatorname{cov}(X_N,X_j) = \operatorname{cov}\left(1-\sum_{i=1}^{N-1}X_i,X_j\right) = -\left(\sum_{i=1}^{N-1}\operatorname{cov}(X_i,X_j)\right) = -\sum_{i=1}^{N-1} C_{i,j},$$ that is, $\Sigma_{N,j}$ is the negative of the sum of the entries in the $j$-th column of $C$. Put another way, the column sums of $\Sigma$ equal $0$, and of you think about it a little, so do the row sums of $\Sigma$ equal $0$.

Note that all that we have used is the fact that covariance is a bilinear function.

  • $\begingroup$ Thank you very much! But I struggle with the point that $X_1,\ldots,X_N$ is also jointly normal distributed...because just sampling from an $N$-Dimensional normal with the given Variance matrix does not automatically fullfill that the draw is part of the $N-1$ dimensional subspace, or does it? $\endgroup$ Commented Oct 25, 2015 at 20:33
  • $\begingroup$ Ah sorry, now I understand it...the bordered matrix is not of full rank, so strictly speaking I would not be able to draw from this distribution..correct? $\endgroup$ Commented Oct 25, 2015 at 20:34
  • $\begingroup$ Yes, the bordered matrix has the same rank as $C$. $\sum_i\sum_j\Sigma_{i,j} = \operatorname{var}\left(\sum_{k=1}^N X_k\right) = 0$. $\endgroup$ Commented Oct 25, 2015 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.