8
$\begingroup$

I am self-learning on linear model theory right now, and one thing I find surprising is that although $\mathbb{E}[\mathbf{Y}]$ is defined for a random vector $\mathbf{Y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n\end{bmatrix}$, there is no mention of further moments besides the covariance matrix.

Google searching hasn't turned up much. Are $k$th (raw) moments of $\mathbf{Y}$ considered, or is there a different idea I don't know about?

I am learning from the text Plane Answers to Complex Questions (the TOC starts in p. 17 of the linked file). By "considered," what I mean is is there such a thing as $\mathbb{E}\left[\mathbf{Y}^k\right]$, and if so, how would such a concept be defined? The book I have only covers the first raw moment, and I find it a bit strange that there is no mention of how to define $\mathbb{E}\left[\mathbf{Y}^k\right]$ given my experience in univariate probability, nor do I have the expertise to define it.

Furthermore, if $\mathbb{E}\left[\mathbf{Y}^k\right]$ isn't defined, is there perhaps a related concept that I don't know about that is used instead?

$\endgroup$
  • $\begingroup$ This page shows the relation between raw and central moments, and describes the advantage of central moments. $\endgroup$ – EdM Jun 1 '15 at 16:34
  • $\begingroup$ @EdM I don't understand; that looks like it's talking about univariate moments, which I am extremely familiar with. I am wondering if there is any consideration of $k$-th raw moments ($k \geq 2$) for the multivariate case (i.e., with random vectors), not the univariate case, and if so, how such a concept would be defined. $\endgroup$ – Clarinetist Jun 1 '15 at 16:37
  • $\begingroup$ Related: mathoverflow.net/questions/202732/… $\endgroup$ – Andrew M Jun 1 '15 at 16:49
  • $\begingroup$ My sense is that statistics invariant with respect to translation along the axis of a variable are considered the most useful, and thus raw moments wouldn't be examined so frequently as central moments, which do have that useful property. This Cross Validated page includes extensive discussion of higher-order moments and related issues. $\endgroup$ – EdM Jun 1 '15 at 16:52
  • $\begingroup$ Could you amplify what you mean by "linear model theory" and "considered"? In some simpler applications of linear models assumptions are made only about the first two moments of $\mathbf{Y}$, but in others--such as generalized linear models--assumptions are made that have specific implications for the entire distribution of $\mathbf{Y}$. $\endgroup$ – whuber Jun 1 '15 at 17:10
8
$\begingroup$

The proper analog of univariate moments in a multivariate setting is to view the exponent $\mathbf{k} = (k_1, k_2, \ldots, k_n)$ as a vector, too. The exponential notation with vector bases and vector exponents is a shorthand for the product,

$$\mathbf{y}^\mathbf{k} = y_1^{k_1} y_2 ^{k_2} \cdots y_n^{k_n}.$$

For any such vector $\mathbf{k}$, the (raw) $\mathbf{k}^\text{th}$ moment of the random variable $\mathbf{Y}$ is defined to be

$$\mu_\mathbf{k} = \mathbb{E}\left(\mathbf{Y}^\mathbf{k}\right).$$

To motivate such a definition, consider a univariate moment of a linear function of $\mathbf{Y}$:

$$\mathbb{E}\left(\left(\lambda_1 Y_1 + \cdots + \lambda_n Y_n\right)^m\right) = \sum_\mathbf{k} \binom{m}{\mathbf{k}}\lambda^\mathbf{k} \mu_\mathbf{k}$$

where the sum occurs over all $\mathbf{k}$ whose components are whole non-negative numbers summing to $m$ and $\binom{m}{\mathbf{k}} = m!/(k_1!k_2!\cdots k_n!)$ are the multinomial coefficients. The appearance of the multivariate moments on the right hand side shows why they are natural and important generalizations of the univariate moments.

These show up all the time. For instance, the covariance between $Y_i$ and $Y_j$ is none other than

$$\text{Cov}(Y_i, Y_j) = \mathbb{E}(Y_i Y_j)- \mathbb{E}(Y_i)\mathbb{E}(Y_j) = \mu_{\mathbf{k}_i + \mathbf{k}_j} - \mu_{\mathbf{k}_i}\mu_{\mathbf{k}_j}$$

where $\mathbf{k}_i$ and $\mathbf{k}_j$ are the indicator vectors with zeros in all but one place and a one in the indicated location. (The same formula elegantly yields the variance of $Y_i$ when $i=j$.)

There are natural generalizations of all univariate moment concepts to the multivariate setting: a moment generating function, cumulants, a cumulant generating function, central moments, a characteristic function, and algebraic and analytical relationships among them all.

Reference

Alan Stuart and J. Keith Ord, Kendall's Advanced Theory of Statistics, Fifth Edition. Oxford University Press, 1987: Volume I, Chapter 3, Moments and Cumulants.

$\endgroup$
3
$\begingroup$

In addition to @whuber's points

1) I am not sure what linear model theory entails but remember that in linear models we are generally dealing with normal random variables which have 0 skew and 0 kurtosis.

2) More generally, the question is of the form "How precise is precise?". If I want to describe IID samples I could say I only want the mean. Alternatively I could say I want the mean and the errors in the means. An even more detailed alternative would be means, errors in the means and errors in the errors in the means. From this pattern you can see how the higher moments keep mounting. There is no real solution to this problem so people generally stop at level 2 (i.e., mean and variance). That is not to say the higher moments are useless. In fact, for problems involving fat-tailed distributions these problems become relevant

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.