Convergence of weighted regression solution to the solution of L1 regression

Question

I have read in this paper that the weighted regression solution converges to the solution of L1 regression,

for weights, $$W_i=1/|y_i-\hat y_i|$$

I worked this out but unfortunately, I lost.

Could anyone provide me with a reference that includes this derivation?

Answer 1

minimize the sum of squared errors $\sum_i \epsilon_i(\theta)^2$ with $\epsilon_i(\theta) = y_i - \mathbf x_i^\top \theta$ which is simply ordinary least squares estimation for $\theta$. But clearly $$\sum_i \epsilon_i(\theta)^2 = \sum_i \lvert\epsilon_i(\theta)\lvert^2$$

now use weights and observe that $\lvert y_i - \hat y_i(\theta)\lvert = \lvert \epsilon_i(\theta)\lvert$ then with weights $w_i=1/\lvert y_i - \hat y_i(\theta)\lvert$ it follows that $$\sum_i \epsilon_i(\theta)^2w_i = \sum_i \lvert\epsilon_i(\theta)\lvert^2w_i= \sum_i \lvert\epsilon_i(\theta)\lvert^2/\lvert y_i - \hat y_i(\theta)\lvert= \sum_i \lvert\epsilon_i(\theta)\lvert^2/\lvert \epsilon(\theta)\lvert =\sum_i \lvert\epsilon_i(\theta)\lvert$$ which is LAD.

Admittedly this in itself does not answer why you get convergence but that should follow from convergence results on Iteratively Reweighted Least Squares (IRLS) see this wiki IRLS on wiki