1
$\begingroup$

The following is a proof of the existence of M-estimates from the book of Maronna et al. "Robust Statistics". While I understand their strategy, I am having trouble comprehending a step on their proof, namely how taking $\left\| \boldsymbol{\beta} \right\| >b$, implies the inequality $\max_i |y_i -\mathbf{x}_i^{\prime} \boldsymbol{\beta} | \geq b_0$. Could you please help me see how one arrives at this condition? I suspect that I am missing either the geometric picture or some intermediate steps which do not come naturally.

Thank you.

enter image description here

$\endgroup$
1
$\begingroup$

This is not a rigorous explanation. I think I understand your confusion. There is very little "regularity" we can assume with arbitrary risk/loss functions. For regular likelihoods, optimization findings are easily available because it's easy to show the score functions are linear in $\beta$. Here, it turns out, existence results are simply a matter of proving that "wilder" and "wilder" choices of $\beta$ will give loss values that spiral into the infinte. Therefore, we can define a ball of $\beta$s for which the risk is continuous and obtain at least one infimum.

$R$ is a function of the residuals. If we define a compact ball for the $\beta$s, the question is: can we produce a similar compact ball for the residuals? If so, we can obtain some optimization results for $R$. Arbitrarily, $2L$ is chosen as a ball for $R$ for which we can indeed construct a ball in $R^p$ for the $\beta$s

It's not written in the proof, but:

$\text{max}_i \left| y_i - x_i \beta \right| \ge ba - \text{max}_i |y_i| \ge b_0$ for all $\| \beta \| > b$

Recall both the $b$ and $a$ were chosen very small in some sense, so that $b_0$ may be the smallest residual. The $b$ is the boundary of a ball in the parameter space, and the $a$ is a minimax like estimator which tells you how "large" a fitted value may be... the choice of $\|\beta\| = 1$ is arbitrary, you see, since any value of $\beta$ in $R^p$ is at least proportional to some $\beta^**b$ where $\|\beta^*\|=1$. Therefore, any choice of value for the norm of $\beta$ (like 3 or 12 instead of 1) would have been proportional to the $a$ we chose. It simply gives you the "biggest" fitted value.

$\endgroup$
  • $\begingroup$ Okay but I still do not get why the inequalities hold. This was my question by the way, I understand the strategy but not their means. $\endgroup$ – JohnK Oct 13 '15 at 0:29
  • $\begingroup$ @JohnK I think it's basically the minimax inequality: $\text{min}_\beta \text{max}_i \| y_i - X_i \beta\| \ge \text{max}_i \min_\beta \|y_i - X_i \beta \|$. Then of course, choosing the $a$ and $b$ as given will give you a lower bound for any possible residual in the ball. Does that help? $\endgroup$ – AdamO Oct 13 '15 at 19:26
  • $\begingroup$ Α little bit. I am sorry but I am utterly confused with their definitions, their indices and their inequalities, it will take some time to grasp the ideas. $\endgroup$ – JohnK Oct 13 '15 at 19:47
  • $\begingroup$ @JohnK I'm working it out too :) minimax estimation, though, is very interesting stuff. Remember the least squares estimator is just $\hat{\beta} = \min_\beta \| y - X \beta \|$ where the norm is the $\mathcal{L}_2$ norm. The interesting bit about minimax is: betas in a neighborhood of that beta are also pretty good relatively speaking. Because of the convexity of the norm operator, if you replace the $\mathcal{L}_2$ norm with the $\mathcal{L}_\infty$ norm (max), you get a nice wide open ball of betas that map continuously into spaces defined by general loss functions, so optimization holds. $\endgroup$ – AdamO Oct 13 '15 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.