Lately I've seen some advantages mostly in model generalization of minimizing an the mean absolute error (or I guess Laplacian MLE would be an equivalent way of saying it). I'm debating first on what options I have to optimize with, and second of all some efficiency considerations for these options. I've made a list below of what I know so far as a starting point to get some feedback on feasibility/efficiency of these methods for this task. If anyone has some solid theory to inform on the advantages/disadvantages of these and any other methods, it would be greatly appreciated! Please consider model size as it grows larger/deeper as well...in that evaluating the model/running for a training iteration may be very expensive (so if a method facilitates analytical gradient-based optimization it would have the advantage of minimizing training time, all else equal).

  1. Smooth approximation to the ABS function such as here or here
  2. Iteratively reweighted least squares was apparently made for L1 norm optimization IRLS Wiki
  3. Quadratic programming, interior point method (not exactly sure how these work or if they're applicable)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.