# Taylor Series and Multivariate Delta Method

I asked this question on http://math.stackexchange.com/ but did not get any answer. Sorry for cross posting.

I'm trying to understand delta method for matrices and vectors to find the variance-covariance matrices for the functions of matrices and vectors. Please see my attempt below. I'm not sure it is right or wrong and especially confused whether it should be $\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\}$ or $\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}^{\prime}\right)}\right\}$. I would highly appreciate experts help in this regard. Thanks

If $\mathbf{X}$ is a matrix then a Taylor series expansion of $f\left(\mathbf{X}\right)$ about $\mathbf{X}_{0}$ is given by:

\begin{align*} f\left(\mathbf{X}\right) & =f\left(\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right)+\left(\begin{array}{c} \textrm{2-nd and higher}\\ \textrm{order term} \end{array}\right)\\ f\left(\mathbf{X}\right) & \approxeq f\left(\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right). \end{align*}

The approximated variance of $f\left(\mathbf{X}\right)$ about $\mathbf{X}_{0}$ can be computed as

\begin{align*} \mathbb{V}\left[f\left(\mathbf{X}\right)\right] & \approxeq\left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[\left.\frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}}\right|_{\mathbf{X}_{0}}\right] \end{align*}

Example

If $f\left(\mathbf{X}\right)=\mathbf{X}^{-1}$ then \begin{align*} \frac{\partial f\left(\mathbf{X}\right)}{\partial\mathbf{X}} & =\frac{\partial\mathbf{X}^{-1}}{\partial\mathbf{X}}=-\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1} \end{align*} Therefore the approximated variance of $f\left(\mathbf{X}\right)=\mathbf{X}^{-1}$ about $\mathbf{X}_{0}$ is \begin{align*} \mathbb{V}\left(\mathbf{X}^{-1}\right) & \approxeq\left[-\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[-\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]\\ & =\left[\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[\left(\mathbf{X}^{\prime}\otimes\mathbf{X}\right)^{-1}\right]. \end{align*}

If $\mathbf{X}$ is a matrix and $\mathbf{y}$ is a column vector then a Taylor series expansion of $f\left(\mathbf{X},\mathbf{y}\right)$ about $\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)$ is given by: \begin{align*} f\left(\mathbf{X},\mathbf{y}\right) & =f\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{y}-\mathbf{y}_{0}\right)+\left(\begin{array}{c} \textrm{2-nd and higher}\\ \textrm{order term} \end{array}\right)\\ f\left(\mathbf{X},\mathbf{y}\right) & \approxeq f\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{X}-\mathbf{X}_{0}\right)+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\left(\mathbf{y}-\mathbf{y}_{0}\right) \end{align*}

The approximated variance of $f\left(\mathbf{X},\mathbf{y}\right)$ about $\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)$ can be computed as

\begin{align*} \mathbb{V}\left[f\left(\mathbf{X},\mathbf{y}\right)\right] & \approxeq\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]+\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right]^{\prime}\mathbf{\mathbb{V}\left(\mathbf{y}\right)}\left[\left.\frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}}\right|_{\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)}\right] \end{align*}

Example

If $f\left(\mathbf{X},\mathbf{y}\right)=\mathbf{X}\mathbf{y}$ then \begin{align*} \frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{X}} & =\frac{\partial\mathbf{X}\mathbf{y}}{\partial\mathbf{X}}=\frac{\partial\textrm{vec}\left(\mathbf{X}\mathbf{y}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}=\frac{\partial\textrm{vec}\left(\mathbf{I}\mathbf{X}\mathbf{y}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}\\ & =\frac{\partial\left(\mathbf{y}^{\prime}\otimes\mathbf{I}\right)\textrm{vec}\left(\mathbf{X}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}=\left(\mathbf{y}^{\prime}\otimes\mathbf{I}\right)\frac{\partial\textrm{vec}\left(\mathbf{X}\right)}{\partial\textrm{vec}\left(\mathbf{X}\right)}\\ & =\left(\mathbf{y}^{\prime}\otimes\mathbf{I}\right)\qquad\textrm{and}\\ \frac{\partial f\left(\mathbf{X},\mathbf{y}\right)}{\partial\mathbf{y}} & =\frac{\partial\mathbf{X}\mathbf{y}}{\partial\mathbf{y}}=\mathbf{X}. \end{align*} Therefore the approximated variance of $f\left(\mathbf{X},\mathbf{y}\right)=\mathbf{X}\mathbf{y}$ about$\left(\mathbf{X}_{0},\mathbf{y}_{0}\right)$ is \begin{align*} \mathbb{V}\left(\mathbf{X}\mathbf{y}\right) & \approxeq\left(\mathbf{\mathbf{y}_{0}}^{\prime}\otimes\mathbf{I}\right)^{\prime}\mathbb{V}\left\{ \mathbf{\textrm{vec}\left(\mathbf{X}\right)}\right\} \left(\mathbf{\mathbf{y}_{0}}^{\prime}\otimes\mathbf{I}\right)+\mathbf{\mathbf{X}}_{0}^{\prime}\mathbf{\mathbb{V}\left(\mathbf{y}\right)}\mathbf{\mathbf{X}}_{0}. \end{align*}

Edited

Let $\mathbf{A}$ be $m\times n$ matrix with $\mathbf{a}_{i}(i=1,\ldots,n)$ as the \emph{$i$}-th column vector. The vectorization operation , $\textrm{vec}\left(.\right)$, is an operation from $\mathcal{R}^{m\times n}$ to $\mathcal{R}^{mn}$, with

$\textrm{vec}(\mathbf{A})=\textrm{vec}\left(\begin{bmatrix}a_{11} & \ldots & a_{1n}\\ \vdots & \ddots & \vdots\\ a_{m1} & \ldots & a_{mn} \end{bmatrix}\right)=\textrm{vec}\left(\left[\begin{array}{ccc} \mathbf{a}_{1} & \cdots & \mathbf{a}_{n}\end{array}\right]\right)=\left[\begin{array}{c} \mathbf{a}_{1}\\ \vdots\\ \mathbf{a}_{n} \end{array}\right].$

-
Could you define $\text{vec}(\mathbf{X})$? – Aniko Jul 20 '12 at 14:23
@Aniko: Please see my edits. Thanks – MYaseen208 Jul 20 '12 at 14:56
I think you go wrong at the very first formula - the Taylor decomposition of a matrix-valued function. Its derivative with respect to a matrix is a 4-dimensional object (tensor), and you can't just go around treating it like a matrix. At the very least you have to very carefully define all those notations, conversions to matrices, etc. For example, I think in that Taylor expansion, you should already have $\text{vec}(\mathbf{X})$ for the dimensions to make any kind of sense. But I don't know much about matrix calculus, just trying to use common sense. – Aniko Jul 20 '12 at 15:45
This looks right. Regarding whether you should have $\mbox{vec}\left(\mathbf{X}\right)$ or $\mbox{vec}\left(\mathbf{X}'\right)$, it depends on how you define the derivative, i.e. in 'numerator layout' or 'denominator layout' (see en.wikipedia.org/wiki/Matrix_calculus#Layout_conventions). – shabbychef Apr 22 '14 at 5:18