Given a covariance matrix $\mathbf{\Sigma} \in \mathbb{R}^{N \times N}$ between $N$ variables of interest, generated by some kernel function $\mathbf{\kappa}(\mathbf{x}, \mathbf{y})$, and some vector $\mathbf{w} \in \mathbb{R}^{N \times 1}$ of normalised weights with one dimension for each given variable, how can one weight the covariance matrix $\mathbf{\Sigma}$?

This is motivated by the notion that when using the covariance matrix $\mathbf{\Sigma}$ in some statistical computation, one may wish to increase/decrease the impact of some variables of interest in the model.

Assume that $\mathbf{\Sigma}$ is given and can not be rebuilt (for example, no access to original data), one might attempt to perform the weighting by generating a diagonal matrix $\mathbf{W}$ of the weight vector $\mathbf{w}$ and proceeding as follows

$\mathbf{\Sigma}_{*} = \mathbf{WX}$. However, in this case $\mathbf{\Sigma}_{*}$ would no longer be a valid covariance matrix (unlikely to be symmetric and positive definite).

However, if one were to take the eigendecomposition of $\mathbf{\Sigma}$, such that $\mathbf{V}$ is the matrix of eigenvectors of $\mathbf{\Sigma}$ and $\mathbf{D}$ the diagonal matrix of eigenvalues, a possible alternative weighting is as follows

$\mathbf{\Sigma}_{*} = \mathbf{V}\big((\mathbf{W} + \mathbf{I}\epsilon)\mathbf{D}\big)\mathbf{V}^{-1}$

for some small $\epsilon > 0$.

The above is an application of the following properties of the eigendecomposition.

$ \mathbf{AV} = \mathbf{VD}\\ \text{and}\\ \mathbf{A} = \mathbf{VDV}^{-1} $

The intuition behind this approach is that by scaling the eigenvalues accordingly, thus increasing/decreasing the scale but not the direction of the basis vectors of $\mathbf{\Sigma}$, one may recover the weighted covariance matrix of interest.

Though this appears to give me a $\mathbf{\Sigma}_{*}$ that has the properties of a valid covariance matrix, will this approach retain the relationships between variables as expected?

  • 1
    $\begingroup$ Pardon me, but what the heck are you trying to do? What is your intended downstream use of whatever you come up with? $\endgroup$ – Mark L. Stone Jul 15 '18 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.