2
$\begingroup$

The definition of an M-estimator is an estimator (from Casella and Berger) of the form $$\hat{\theta}=\min \sum_{i=1}^n \rho(X_i-\theta),$$

where $X_1,X_2, \cdots, X_n$ is the data for some function $\rho$.

The LASSO estimate is defined as $$\hat{\theta}=\min ||AX-\theta||_2^2 + \lambda ||\theta||_1,$$ for some matrix A, and a vector parameter $\theta$.

Many papers refer to LASSO as an M-estimator, which does not appear to be true. Am I missing something here? The definition seems to imply there can not be any M-estimator for a vector parameter.

$\endgroup$
2
$\begingroup$

I am not familiar with LASSO being referred to as a conventional M-Estimator, as the conventional M-Estimator does not have a constraint, leading to familiar unconstrained optimization problems. I think LASSO is generally referred to as a regularized M-Estimator, i.e. an M-estimator with a regularity parameter. This would be $\lambda$ on the $\mathcal{L}_1$ norm or (LASSO) $\mathcal{L}_2$ norm (Ridge). See here for a nice description of the problem.

$\endgroup$
  • $\begingroup$ Thanks, but even $\min ||AX-\theta||_2^2$$ does not appear to be an M-estimator by the definition, right? $\endgroup$ – Devil Sep 11 '14 at 20:28
  • 2
    $\begingroup$ The definition you supply in the question doesn't appear to be correct. That would make quasilikelihood and GEE not a form of M estimation which isn't the case. $\endgroup$ – AdamO Sep 11 '14 at 22:12
  • $\begingroup$ You are right. Many other papers like www4.stat.ncsu.edu/~boos/papers/mest6.pdf seem to define an M-estimator differently. They define an M-estimator as an estimator obtained by solving equations of the form $\sum_{i=1}^n \psi(X_i,\theta)=0$ as an M-estimator, in which case even least squares is an M-estimator. Thank you. $\endgroup$ – Devil Sep 12 '14 at 18:27
  • $\begingroup$ Yes, this is more cohesive with my knowledge. The solution to an optimization problem (as in the case of the argmin definition) boils down to a linear root-finding problem (for "nice" problems) and well suited to Newton Raphson optimization. Optimizing over a general space can be very mean... however regularized M-estimators behave nicely because it is a type of dynamic optimization that can be solved with Lagrange multipliers. $\endgroup$ – AdamO Sep 12 '14 at 22:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.