Return to Question

I need coefficients of autoregressive model of $x$: matrix $B$ with zero diagonal which minimizes squared error:

$$\text{argmin}_B E[||x - Bx||^2]$$

For non-singular covariance, I can get solution directly from $E[xx']^{-1}$

What should I do for singular $E[xx']$?

For instance, suppose

$$X=\left( \begin{array}{cccc} 2 & -1 & -2 & 1 \\ -2 & 1 & -2 & 1 \\ 2 & 1 & 1 & 3 \\ \end{array} \right)$$

I can turn this into 4 regression problems, solve each problem using $\beta=(X'X)^{-1}(Y'X)$, and get following coefficients of auto-regressive model with $x=Bx$ $$B=\left( \begin{array}{cccc} 0 & -\frac{1}{2} & \frac{7}{4} & \frac{7}{8} \\ -2 & 0 & \frac{7}{2} & \frac{7}{4} \\ \frac{4}{7} & \frac{2}{7} & 0 & -\frac{1}{2} \\ \frac{8}{7} & \frac{4}{7} & -2 & 0 \\ \end{array} \right)$$

1. This requires 4 matrix inversions, whereas for non-singular covariance matrix I had to only do 1

2. This is a well-posed problem, but in general it may be ill-posed, so one would use some linear solver instead of $(X'X)^{-1}(Y'X)$ formula. However, calling linear solver for each column is too expensive.

Edit

Aksakal suggested to use regularized estimator for inverting the covariance, trying this out here

Using shrunk covariance estimator with very small lambda gives good results. Numerically inverting singular covariance without regularization and sticking it into the formula gives slightly better error.

Error direct:  1.4051584874249273e-30
Error inverse:  3.2047474274603605e-30
Error Ledoit-Wolf:  9.351834226497054
Error shrunk covariance:  0.7203818230423954
Error very-shrunk covariance:  0.00952058588215488

I need coefficients of autoregressive model of $x$: matrix $B$ with zero diagonal which minimizes squared error:

$$\text{argmin}_B E[||x - Bx||^2]$$

For non-singular covariance, I can get solution directly from $E[xx']^{-1}$ using following procedure:

1. Let $D2$ be the inverse second moment with off-diagonal terms set to zero $$D2=E[xx']^{-1}_d$$

2. Read coefficients $\{a_1\ldots a_n\}$ for the model $x_i=a_0 x_0 + \ldots a_n x_n$ from $i$th row of following matrix

$$B=I-(X'X)^{-1} D2^{-1}$$

What should I do for singular $E[xx']$?

For instance, suppose

$$X=\left( \begin{array}{cccc} 2 & -1 & -2 & 1 \\ -2 & 1 & -2 & 1 \\ 2 & 1 & 1 & 3 \\ \end{array} \right)$$

I can turn this into 4 regression problems, solve each problem using $\beta=(X'X)^{-1}(Y'X)$, and get following coefficients of auto-regressive model with $x=Bx$ $$B=\left( \begin{array}{cccc} 0 & -\frac{1}{2} & \frac{7}{4} & \frac{7}{8} \\ -2 & 0 & \frac{7}{2} & \frac{7}{4} \\ \frac{4}{7} & \frac{2}{7} & 0 & -\frac{1}{2} \\ \frac{8}{7} & \frac{4}{7} & -2 & 0 \\ \end{array} \right)$$

1. This requires 4 matrix inversions, whereas for non-singular covariance matrix I had to only do 1

2. This is a well-posed problem, but in general it may be ill-posed, so one would use some linear solver instead of $(X'X)^{-1}(Y'X)$ formula. However, calling linear solver for each column is too expensive.

Edit

Aksakal suggested to use regularized estimator for inverting the covariance, trying this out here

Using shrunk covariance estimator with very small lambda gives good results. Numerically inverting singular covariance without regularization and sticking it into the formula gives slightly better error.

Error direct:  1.4051584874249273e-30
Error inverse:  3.2047474274603605e-30
Error Ledoit-Wolf:  9.351834226497054
Error shrunk covariance:  0.7203818230423954
Error very-shrunk covariance:  0.00952058588215488

I need coefficients of autoregressive model of $x$: matrix $B$ with zero diagonal which minimizes squared error:

$$\text{argmin}_B E[||x - Bx||^2]$$

For non-singular covariance, I can get solution directly from inverse of $E[xx']^{-1}$

What should I do for singular $E[xx']$?

For instance, suppose

$$X=\left( \begin{array}{cccc} 2 & -1 & -2 & 1 \\ -2 & 1 & -2 & 1 \\ 2 & 1 & 1 & 3 \\ \end{array} \right)$$

I can turn this into 4 regression problems, solve each problem using $\beta=(X'X)^{-1}(Y'X)$, and get following coefficients of auto-regressive model with $x=Bx$ $$B=\left( \begin{array}{cccc} 0 & -\frac{1}{2} & \frac{7}{4} & \frac{7}{8} \\ -2 & 0 & \frac{7}{2} & \frac{7}{4} \\ \frac{4}{7} & \frac{2}{7} & 0 & -\frac{1}{2} \\ \frac{8}{7} & \frac{4}{7} & -2 & 0 \\ \end{array} \right)$$

1. This requires 4 matrix inversions, whereas for non-singular covariance matrix I had to only do 1

2. This is a well-posed problem, but in general it may be ill-posed, so one would use some linear solver instead of $(X'X)^{-1}(Y'X)$ formula. However, calling linear solver for each column is too expensive.

Edit

Aksakal suggested to use regularized estimator for inverting the covariance, trying this out here

Using shrunk covariance estimator with very small lambda gives good results. Numerically inverting singular covariance without regularization and sticking it into the formula gives slightly better error.

Error direct:  1.4051584874249273e-30
Error inverse:  3.2047474274603605e-30
Error Ledoit-Wolf:  9.351834226497054
Error shrunk covariance:  0.7203818230423954
Error very-shrunk covariance:  0.00952058588215488

I need coefficients of autoregressive model of $x$: matrix $B$ with zero diagonal which minimizes squared error:

$$\text{argmin}_B E[||x - Bx||^2]$$

For non-singular covariance, I can get solution directly from $E[xx']^{-1}$

What should I do for singular $E[xx']$?

For instance, suppose

$$X=\left( \begin{array}{cccc} 2 & -1 & -2 & 1 \\ -2 & 1 & -2 & 1 \\ 2 & 1 & 1 & 3 \\ \end{array} \right)$$

I can turn this into 4 regression problems, solve each problem using $\beta=(X'X)^{-1}(Y'X)$, and get following coefficients of auto-regressive model with $x=Bx$ $$B=\left( \begin{array}{cccc} 0 & -\frac{1}{2} & \frac{7}{4} & \frac{7}{8} \\ -2 & 0 & \frac{7}{2} & \frac{7}{4} \\ \frac{4}{7} & \frac{2}{7} & 0 & -\frac{1}{2} \\ \frac{8}{7} & \frac{4}{7} & -2 & 0 \\ \end{array} \right)$$

1. This requires 4 matrix inversions, whereas for non-singular covariance matrix I had to only do 1

2. This is a well-posed problem, but in general it may be ill-posed, so one would use some linear solver instead of $(X'X)^{-1}(Y'X)$ formula. However, calling linear solver for each column is too expensive.

Edit

Aksakal suggested to use regularized estimator for inverting the covariance, trying this out here

Using shrunk covariance estimator with very small lambda gives good results. Numerically inverting singular covariance without regularization and sticking it into the formula gives slightly better error.

Error direct:  1.4051584874249273e-30
Error inverse:  3.2047474274603605e-30
Error Ledoit-Wolf:  9.351834226497054
Error shrunk covariance:  0.7203818230423954
Error very-shrunk covariance:  0.00952058588215488

I need coefficients of autoregressive model of $x$: matrix $B$ with zero diagonal which minimizes squared error:

$$\text{argmin}_B E[||x - Bx||^2]$$

For non-singular covariance, I can get solution directly from inverse of $E[xx']^{-1}$

What should I do for singular $E[xx']$?

For instance, suppose

$$X=\left( \begin{array}{cccc} 2 & -1 & -2 & 1 \\ -2 & 1 & -2 & 1 \\ 2 & 1 & 1 & 3 \\ \end{array} \right)$$

I can turn this into 4 regression problems, solve each problem using $\beta=(X'X)^{-1}(Y'X)$, and get following coefficients of auto-regressive model $Bx$ $$B=\left( \begin{array}{cccc} 0 & -\frac{1}{2} & \frac{7}{4} & \frac{7}{8} \\ -2 & 0 & \frac{7}{2} & \frac{7}{4} \\ \frac{4}{7} & \frac{2}{7} & 0 & -\frac{1}{2} \\ \frac{8}{7} & \frac{4}{7} & -2 & 0 \\ \end{array} \right)$$

1. This requires 4 matrix inversions, whereas for non-singular covariance matrix I had to only do 1

2. This is a well-posed problem, but in general it may be ill-posed, so one would use some linear solver instead of $(X'X)^{-1}(Y'X)$ formula. However, calling linear solver for each column is too expensive.

Edit

Aksakal suggested to use regularized estimator for inverting the covariance, trying this out here . Using shrunk covariance estimator with very small lambda gives good results. Numerically inverting singular matrix without regularization gives model $B$ with slightly better residual.

Error direct:  1.4051584874249273e-30
Error inverse:  3.2047474274603605e-30
Error Ledoit-Wolf:  9.351834226497054
Error shrunk covariance:  0.7203818230423954
Error very-shrunk covariance:  0.00952058588215488

I need coefficients of autoregressive model of $x$: matrix $B$ with zero diagonal which minimizes squared error:

$$\text{argmin}_B E[||x - Bx||^2]$$

For non-singular covariance, I can get solution directly from inverse of $E[xx']^{-1}$

What should I do for singular $E[xx']$?

For instance, suppose

$$X=\left( \begin{array}{cccc} 2 & -1 & -2 & 1 \\ -2 & 1 & -2 & 1 \\ 2 & 1 & 1 & 3 \\ \end{array} \right)$$

I can turn this into 4 regression problems, solve each problem using $\beta=(X'X)^{-1}(Y'X)$, and get following coefficients of auto-regressive model with $x=Bx$ $$B=\left( \begin{array}{cccc} 0 & -\frac{1}{2} & \frac{7}{4} & \frac{7}{8} \\ -2 & 0 & \frac{7}{2} & \frac{7}{4} \\ \frac{4}{7} & \frac{2}{7} & 0 & -\frac{1}{2} \\ \frac{8}{7} & \frac{4}{7} & -2 & 0 \\ \end{array} \right)$$

1. This requires 4 matrix inversions, whereas for non-singular covariance matrix I had to only do 1

2. This is a well-posed problem, but in general it may be ill-posed, so one would use some linear solver instead of $(X'X)^{-1}(Y'X)$ formula. However, calling linear solver for each column is too expensive.

Edit

Aksakal suggested to use regularized estimator for inverting the covariance, trying this out here 

Using shrunk covariance estimator with very small lambda gives good results. Numerically inverting singular covariance without regularization and sticking it into the formula gives slightly better error.

Error direct:  1.4051584874249273e-30
Error inverse:  3.2047474274603605e-30
Error Ledoit-Wolf:  9.351834226497054
Error shrunk covariance:  0.7203818230423954
Error very-shrunk covariance:  0.00952058588215488