Robust regularized regression

Question

I've been using elastic net implemented in R (via glmnet) for some modeling, but I was wondering, due to the number of outliers in my data, if there was some sort of modeling approach for regularized robust regression? e.g. something like elastic net applied to robust regression. if there's something already in R, even better. Just curious to know what's out there.

Answer 1

There seem to be some relevant papers. You can start with:

http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ejs/1316092867

http://arxiv.org/abs/0811.1790

www.econ.kuleuven.be/public/ndbae06/PDF.../sparseLTS.pdf

http://www.google.cl/url?sa=t&rct=j&q=&esrc=s&source=web&cd=20&cad=rja&ved=0CGYQFjAJOAo&url=http%3A%2F%2Fwww.econ.kuleuven.be%2Fpublic%2Fndbae06%2FPDF-FILES%2FsparseLTS.pdf&ei=pSJkUJTiNYug8gTX8IDwAw&usg=AFQjCNFoKj1zoHwYIcXJDbLhUYHYDAscYw&sig2=btUee_XNA9tXs0MsJw_DNA

... but I don't know about implementations.

Answer 2

Suppose we are using L1 norm on error term, the objective is still convex.

The objective is

$$\|Ax-b\|_1+\lambda_2\|x\|^2_2+\lambda_1\|x\|_1$$

Gradient is

$$A^T \text{sign}(Ax-b)+2\lambda_2x+\lambda_1 \text{sign}(x)$$

Here is a R implementation

f<-function(x,A,b,l1,l2){
  e=A %*% x - b
  # v=crossprod(e)+l2*crossprod(x)+l1*sum(abs(x))
  v=sum(abs(e))+l2*crossprod(x)+l1*sum(abs(x))
  return(c(v))
}

gr<-function(x,A,b,l1,l2){
  v=t(A) %*% sign(A %*% x -b)
  return(c(v)+2*l2*x+l1*sign(x))
}

set.seed(0)
par(cex=1.3)
n_data=20
n_feature=2

A=matrix(runif(n_data*n_feature),ncol=n_feature)
b=matrix(runif(n_data),ncol=1)

l1=0.01
l2=0.01

opt=optimx::optimx(runif(n_feature),f, gr,A=A,b=b,l1=l1,l2=l2, method="BFGS")
opt

Here is the results and visualization

> opt
            p1        p2    value fevals gevals niter convcode  kkt1 kkt2 xtimes
BFGS 0.2600659 0.7781711 4.995662    179     29    NA        0 FALSE   NA      0