Consider a non-linear least squares estimation problem, where we use a kernel (i.e. M-estimator) to reduce sensitivity to outliers,e.g.
where $r_k$ are residuals, and $\rho_k$ is a robust kernel.
There are many ways to approach this problem, for example iteratively reweighted least squares (IRLS), or kernel lifting approaches. While I have a basic understanding of the pros and cons of such methods (e.g. IRLS computes the weights at the current linearization point, which is not the best possible approximation), I am unsure about how the robust kernel itself should be chosen.
In most of the papers I've encountered in the computer vision literature, the justification for a particular robust kernel $\rho_k$ goes a little bit like this:
- Obviously setting $\rho_k$ to L1 or L2 norms is a bad idea
- We experimentaly compared every available M-estimaror and chose e.g. Geman-McClure.
From my own experience, it seems to me that the actual form of the chosen kernel $\rho_k$ doesn't really matter, as long as it is linear or quadratic for small residuals and gets nearly flat after a certain threshold (that can be set using statistics from the data), but I suspect that this is more a particularity of the data I've worked with, and not a general rule.
What is the sound way to chose a robust kernel? What are the theoretical arguments that link this to properties of the dataset (references to papers are extremely welcome)?