I have read in this paper that the weighted regression solution converges to the solution of L1 regression,

for weights, $$W_i=1/|y_i-\hat y_i|$$

I worked this out but unfortunately, I lost.

Could anyone provide me with a reference that includes this derivation?


minimize the sum of squared errors $\sum_i \epsilon_i(\theta)^2$ with $\epsilon_i(\theta) = y_i - \mathbf x_i^\top \theta$ which is simply ordinary least squares estimation for $\theta$. But clearly $$\sum_i \epsilon_i(\theta)^2 = \sum_i \lvert\epsilon_i(\theta)\lvert^2$$

now use weights and observe that $\lvert y_i - \hat y_i(\theta)\lvert = \lvert \epsilon_i(\theta)\lvert$ then with weights $w_i=1/\lvert y_i - \hat y_i(\theta)\lvert$ it follows that $$\sum_i \epsilon_i(\theta)^2w_i = \sum_i \lvert\epsilon_i(\theta)\lvert^2w_i= \sum_i \lvert\epsilon_i(\theta)\lvert^2/\lvert y_i - \hat y_i(\theta)\lvert= \sum_i \lvert\epsilon_i(\theta)\lvert^2/\lvert \epsilon(\theta)\lvert =\sum_i \lvert\epsilon_i(\theta)\lvert$$ which is LAD.

Admittedly this in itself does not answer why you get convergence but that should follow from convergence results on Iteratively Reweighted Least Squares (IRLS) see this wiki IRLS on wiki

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.