3
$\begingroup$

I currently have 4 data points and the following Jacobian matrix $A$ and cost vector $b$

$ A = \begin{bmatrix} -0.7867 & 0.0464& -0.6155 & 1.0000 \\ -0.3751 & 0.4299 & -0.8213 & 1.0000 \\ 0.0447 & 0.4895 & -0.8708 & 1.0000 \\ -0.5946 & 0.8029 & 0.0424 & 1.0000 \\ \end{bmatrix} $

$b = \begin{bmatrix} 26.3019 \\ 1.1 \\ 4.4677 \\ 4.6455 \\ \end{bmatrix}$

I am trying to use weighted least squares with the following equation to reduce the effect of the 1st data point.

$H = (A^TWA)^{-1}A^TW$ so that $y = Hb$

However my weighted Jacobian matrix $H$ seems to be virtually the same regardless of what weights I put in.

$W_1 =\begin{bmatrix} 1 & 0& 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \\ \end{bmatrix}$

$H_1 =\begin{bmatrix} 0.4964 & -3.4382& 2.8939& 0.0479\\ -2.4944& 4.2279 & -2.2013& 0.4678\\ 1.2034& -3.8577& 1.6862& 0.9681\\ 2.2469& -5.2755& 3.4167& 0.6119\\ \end{bmatrix}$

$W_2 =\begin{bmatrix} .0001 & 0& 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1 \\ \end{bmatrix}$

$H_2 =\begin{bmatrix} 0.4964 & -3.4382& 2.8939& 0.0479\\ -2.4944& 4.2279 & -2.2013& 0.4678\\ 1.2034& -3.8577& 1.6862& 0.9681\\ 2.2469& -5.2755& 3.4167& 0.6119\\ \end{bmatrix}$

Is there a reason why my weights have no effect on the Jacobian and are there requirements on the Jacobian matrix or the weights of Weighted Least Squares? For the weight matrix $W$, should it be $w^Tw$ for a 4x1 weight vector instead of a diagonal matrix?

$\endgroup$
4
$\begingroup$

This is simply because $A$ is a square and non-singular matrix. Therefore, \begin{align} H = (A^TWA)^{-1}A^TW = A^{-1}(A^TW)^{-1}A^TW = A^{-1}. \end{align} In practice, $A$ is typically slim rectangular ($n\times p$ with $n > p$) so $(A^TWA)^{-1}$ cannot be written to $A^{-1}(A^TW)^{-1}$ thus $W$ would have effect.

$\endgroup$
  • $\begingroup$ As a follow-up is there a way to weigh scenarios where you have square matrices? $\endgroup$ – d.mc2 Jul 31 '17 at 19:07
  • 1
    $\begingroup$ As long as the design matrix is non-singular, then as the calculation in the answer shows, there is no way (the effect of $W$ cancelled with each other). $\endgroup$ – Zhanxiong Jul 31 '17 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.