I've been using elastic net implemented in R (via glmnet) for some modeling, but I was wondering, due to the number of outliers in my data, if there was some sort of modeling approach for regularized robust regression? e.g. something like elastic net applied to robust regression. if there's something already in R, even better. Just curious to know what's out there.
-
$\begingroup$ Have a look at the answers to this similar questions. $\endgroup$– user603May 25, 2016 at 20:39
2 Answers
There seem to be some relevant papers. You can start with:
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ejs/1316092867
http://arxiv.org/abs/0811.1790
www.econ.kuleuven.be/public/ndbae06/PDF.../sparseLTS.pdf
... but I don't know about implementations.
Suppose we are using L1 norm on error term, the objective is still convex.
The objective is
$$\|Ax-b\|_1+\lambda_2\|x\|^2_2+\lambda_1\|x\|_1$$
Gradient is
$$A^T \text{sign}(Ax-b)+2\lambda_2x+\lambda_1 \text{sign}(x)$$
Here is a R implementation
f<-function(x,A,b,l1,l2){
e=A %*% x - b
# v=crossprod(e)+l2*crossprod(x)+l1*sum(abs(x))
v=sum(abs(e))+l2*crossprod(x)+l1*sum(abs(x))
return(c(v))
}
gr<-function(x,A,b,l1,l2){
v=t(A) %*% sign(A %*% x -b)
return(c(v)+2*l2*x+l1*sign(x))
}
set.seed(0)
par(cex=1.3)
n_data=20
n_feature=2
A=matrix(runif(n_data*n_feature),ncol=n_feature)
b=matrix(runif(n_data),ncol=1)
l1=0.01
l2=0.01
opt=optimx::optimx(runif(n_feature),f, gr,A=A,b=b,l1=l1,l2=l2, method="BFGS")
opt
Here is the results and visualization
> opt
p1 p2 value fevals gevals niter convcode kkt1 kkt2 xtimes
BFGS 0.2600659 0.7781711 4.995662 179 29 NA 0 FALSE NA 0