10
$\begingroup$

Given i.i.d. samples from a gaussian distribution $X_1,...,X_n \sim N(\mu,\sigma) $ and the M-estimator, $\mu_m = \underset{a}{\operatorname{argmin}} \sum\rho(|X_i-a|)$, what properties on $\rho$ are sufficient to guarantee $\mu_m \rightarrow \mu$ in probability? Is $\rho$ being strictly convex and strictly increasing sufficient?

$\endgroup$
  • $\begingroup$ Since you may take $\rho(x) = x$ and then $\mu_m$ is the sample mean, that means it could be even not strictly convex, but strictly increasing yes, thus... I would put either strictly convex or strictly increasing, both seems to be sufficient, though have still to prove this. Intuitively strict convexity ensures unique global minimum, for strictly increasing it's the gaussianity assumption that matters. $\endgroup$ – Dmitrij Celov Aug 3 '11 at 12:13
1
$\begingroup$

The paper Asymptotics for minimisers of convex processes by Hjort and Pollard may help here, although it does not specialize to Gaussian distributions, and it considers a more general form of contrast function, namely $\rho(x,a)$, though their notation is $g(y,t)$. In addition to convexity of $g$ in $t$, they require an expansion of $g$ in $t$ around $\theta_0$, in a certain sense that's related to the data distribution. So, not as simple as just saying $\rho$ is convex or increasing, but perhaps if you restrict the theorem to Gaussian distributions and $g$ to have the form you specify, you can get an even neater set of conditions. I'll rewrite their theorem here for completeness, slightly paraphrased:

Suppose we have

  • $Y,Y_{1},Y_{2},\ldots$ i.i.d. from distribution $F$
  • Parameter of interest $\theta_{0}=\theta(F)\in\cal{R}^{p}$
  • $\theta_{0}\in\arg\min_{t\in\cal{R}^{p}}\mathbb{E} g(Y,t)$, where $g(y,t)$ is convex in $t$.
  • We have a "weak expansion" of $g(y,t)$ in $t$ around $\theta_{0}$: $$ g(y,\theta_{0}+t)-g(y,\theta_{0})=D(y)^{T}t+R(y,t), $$ for a $D(y)$ with mean zero under $F$ and $$ \mathbb{E} R(Y,t)=\frac{1}{2}t^{T}Jt+o(\left|t\right|^{2}),\mbox{ as }t\to0 $$ for a positive definite matrix $J$.
  • $\mbox{Var}[ R(Y,t) ]=o(\left|t\right|^{2})$ as $t\to0$.
  • $D(Y)$ has a finite covariance matrix $K=\int D(y)D(y)^{T}\, dF(y)$.

THEN any estimator $\hat{\theta}_{n}\in\arg\min_{\theta\in\cal{R}^{p}}\sum_{i=1}^{n}g(Y_{i},t)$ is $\sqrt{n}$-consistent for $\theta_{0}$, and asymptotically normal with $$ \sqrt{n}\left(\hat{\theta}_{n}-\theta_{0}\right)\stackrel{d}{\rightarrow}\mathcal{N}_{p}(0,J^{-1} K J^{-1}). $$

$\endgroup$
0
$\begingroup$

This will not be an answer, since it will reduce your problem to another one, but I think it might be useful. Your question is basically about consistency of M-estimator. So first we can look at the general results. Here is the result from van der Vaart book (theorem 5.7, page 45):

Theorem Let $M_n$ be random functions and let $M$ be a fixed function of $\theta$ such that for every $\varepsilon>0$

$$\sup_{\theta\in\Theta}|M_n(\theta)-M(\theta)|\xrightarrow{P}0,$$

$$\sup_{\theta:d(\theta,\theta_0)\ge\varepsilon} M(\theta)<M(\theta_0).$$

Then any sequence of estimators $\hat\theta_n$ with $M_n(\hat\theta_n)\ge M_n(\theta_0)-o_P(1)$ converges in probability to $\theta_0$

In your case $\theta_0=\mu$, $M(\theta)=E\rho(|X-\theta|)$ and $M_n(\theta)=\frac{1}{n}\sum \rho(|X_i-\theta|)$

The key condition here is the uniform convergence. In page 46 van der Vaart says

that for averages which is your case this condition is equivalent to set of functions $\{m_\theta, \theta\in\Theta\}$ ($m_\theta=\rho(|x-\theta|)$ in your case) being Glivenko-Canteli. One simple set of sufficient conditions is that $\Theta$ be compact, that the functions $\theta\to m_\theta(x)$ are continuous for every $x$, and that > they are dominated by an integrable function.

In Wooldridge this result is formulated as theorem called Uniform Weak Law of Large Numbers, page 347 (first edition), theorem 12.1. It only adds measurability requirements to what van der Vaart states.

In your case you can safely pick $\Theta=[\mu-C,\mu+C]$ for some $C$, so you need to show that there exists function $b$ such that

$$|\rho(|x-\theta|)|\le b(x)$$

for all $\theta\in\Theta$, such that $Eb(X)<\infty$. Convex function theory might be of help here, since you basicaly can take

$$b(x)=\sup_{\theta\in\Theta}|\rho(|x-\theta|)|.$$

If this function has nice properties then you are good to go.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.