9
$\begingroup$

I am reading a paper which says that LASSO is asymptotically biased while SCAD is not. I take asymptotic (un)biasedness to concern the slope estimators from LASSO and SCAD as the sample size goes to infinity, but I am not sure. I wonder what exactly these statements mean, under which assumptions they hold and whether these assumptions are realistic.

In my understanding, LASSO is asymptotically biased given fixed regularization intensity $\lambda$. However, realistically $\lambda$ would not be kept fixed as the sample size grows but would rather be reduced (e.g. this would be the case if one used LOOCV for selecting $\lambda$, something that is fairly common), reducing the bias accordingly. Taking this to the limit, it appears LASSO would not be asymptotically biased.

On the other hand, if we look at some typical pictures illustrating LASSO and SCAD estimators, they often consider them as functions of the slope coefficient. There, we see that LASSO is asympt. biased while SCAD is asympt. unbiased when the slope coefficient (rather than the sample size) goes to infinity. (See below.) Again, I think $\lambda$ is fixed here.

enter image description here

So I am confused, and hence my question.

Update 1: Here is a small simulation compatible with the statement that LASSO is asymptotically biased. $\lambda$ is selected via 8-fold CV.
(A simulation where $\lambda$ is selected by LOOCV yields similar results for sample sizes up to $2^{15}$; larger sample sizes are not feasible on my laptop.)

enter image description here

The figure corresponds to Setup no. 2 in the code: one relevant regressor, one irrelevant regressor. The top figure is the bias of the coefficient on the relevant regressor, the bottom figure is the bias of the coefficient on the irrelevant regressor. As you can see, the bias of the relevant regressor shrinks initially with the sample size $n$ but stops and remains nonzero after $n$ gets sufficiently large. (It may be a bit difficult to see from the graph, but the bias indeed stops shrinking for the first regressor after $n=2^{13}$.)

library(glmnet)

m=1e3 # number of simulation runs
ns=2^c(5:17) # sample sizes
k=length(ns)

bias1=rep(NA,k); names(bias1)=ns; bias2=bias1

for(j in 1:k){
 n=ns[j]; print(paste0(Sys.time()," n = ",n))
 beta1_hat=beta2_hat=rep(NA,m)
  for(i in 1:m){
   set.seed(i); data=matrix(rnorm(3*n),ncol=3); x1=data[,1]; x2=data[,2]; eps=data[,3]
   # Choose one of the 3 lines below:
   #beta1=0; beta2=0 # Setup no. 1: both regressors irrelevant
   beta1=1; beta2=0 # Setup no. 2: only the first regressor relevant
   #beta1=1; beta2=1 # Setup no. 3: both regressors relevant
   y=beta1*x1+beta2*x2+eps # zero mean for simplicity
   cvfit=cv.glmnet(x=cbind(x1,x2),y=y,nfolds=8)
   coef=coef(cvfit,s="lambda.min")
   beta1_hat[i]=coef[2]
   beta2_hat[i]=coef[3]
  }
 bias1[j]=mean(beta1_hat)-beta1
 bias2[j]=mean(beta2_hat)-beta2
}; print(paste0(Sys.time()," The end"))

par(mfrow=c(2,1),mar=c(4,4,2,0.5))
 at=c(1:k); labels=paste0("2^",log(ns,2))
 ylim=range(bias1,-bias1); plot(bias1,ylim=ylim,xaxt="n",xlab="sample size"); axis(side=1,at=at,labels=labels); abline(h=0)
 ylim=range(bias2,-bias2); plot(bias2,ylim=ylim,xaxt="n",xlab="sample size"); axis(side=1,at=at,labels=labels); abline(h=0)
par(mfrow=c(1,1))

Update 2: searching for "asymptotically biased" online yields Javanmard & Montanari "Confidence intervals and hypothesis testing for high-dimensional regression" (2014) where the matter is discussed in the first 12 pages of the document, especially Theorems 6-8 and some discussion that follows in Section 2.2. The material is unfortunately highly technical and presented using fairly complicated notation. Figuring it out by myself is quite a challenge.

$\endgroup$
4
  • 1
    $\begingroup$ which paper are you reading? $\endgroup$
    – user257566
    Jul 18, 2021 at 3:37
  • 1
    $\begingroup$ @user257566, it is a paper I have recently been reviewing. It has not been published yet. $\endgroup$ Jul 18, 2021 at 5:26
  • $\begingroup$ A bounty has been wasted on this question as it still lacks a satisfactory answer. I will consider awarding another bounty if you (the reader) can answer the question better. $\endgroup$ Aug 15, 2021 at 18:03
  • $\begingroup$ Sidenote: Because of the symmetry of the problem the bias for the parameter 2 should be zero, even for small sample sizes. What you see in the second graph is the error due to the variance. $\endgroup$ Dec 23, 2022 at 9:45

3 Answers 3

1
$\begingroup$

If you are increasing the length of the vectors, then LASSO is asymptotic unbiased because the regularisation intensity $\lambda$ decreases as the sample increases.

Intuitively: You can get effectively the same asymptotic behaviour by keeping the same size of the vector length, but reduce instead the magnitude of the noise term eps. As the sample size increases the means of the validation sets get closer to the true values, and the cross validation procedure should make the LASSO a consistent estimator (get the error as small as you like by adding more data) which implies that it is also asymptotic unbiased.

Possibly the bias in your example is because the glmnet algorithm has a limit for the $\lambda$. It performs in a range from $10^{-4}\lambda^\star$ to $\lambda^\star$ where $\lambda^\star$ is the smallest value for which all coefficients are non-zero.

$\endgroup$
4
  • $\begingroup$ Very interesting! Let me try running the code with a different range of lambdas. What would you suggest instead of $10^{-4}$ to make the bias go away? (I suppose it would be smarter to decrease the magnitude of eps instead, but for simplicity w.r.t. the existing code I will just change the lambda grid first.) Also, I am still interested in the question in bold, i.e. trying to understand what people mean when they say LASSO is asympt. biased. $\endgroup$ Dec 23, 2022 at 10:57
  • $\begingroup$ @RichardHardy there's multiple ways how people can consider asymptotic bias. To simulate the graph with the three curves for LASSO, SCAD, MCP, you change the true beta while keeping other things the same. Another way how LASSO is biased, even asymptotic behaviour, is when $\lambda$ is kept fixed and no cross validation is performed. I believe that this bias occurs in the same way for other methods. $\endgroup$ Dec 23, 2022 at 13:33
  • $\begingroup$ Edit, to simulate that graph with the three curves you plot the regularised $\beta$ as function of the OLS solution. The graph is described in "Adaptive lasso, MCP, and SCAD" by Patrick Breheny. The graph shows the closed form solutions that can be made when the regressors are orthogonal. The amount of bias is for all cases dependent on the $\lambda$, so if you include the effect of cross validation then the bias will diminish for all cases, also LASSO. $\endgroup$ Dec 23, 2022 at 13:59
  • $\begingroup$ Well, perhaps it is so. That would explain it. Now, I have tried changing the lambda grid in the simulation but still got the bias. I will have to experiment more, but if you have a ready solution that makes the bias go away, I would be curious to learn it. $\endgroup$ Dec 23, 2022 at 14:40
0
$\begingroup$

In my understanding, LASSO is asymptotically biased given fixed regularization intensity λ. However, realistically λ would not be kept fixed as the sample size grows but would rather be reduced ... reducing the bias accordingly. Taking this to the limit, it appears LASSO would not be asymptotically biased.

I believe you're misunderstanding what "fixed regularization intensity" is and what impact it has.

Lasso is biased because it penalizes all model coefficients with the same intensity. A large coefficient and a small coefficient are shrunk at the same rate. This biases estimates of large coefficients which should remain in the model. Under specific conditions, the bias of large coefficients is $\lambda$ (slide 2).

Variable selection methods which shrink large coefficients more slowly than small coefficients avoid this problem and may produce unbiased estimates. SCAD is one example. Adaptive Lasso is another.

$\endgroup$
7
  • $\begingroup$ Thank you for you answer! Slide 2 claims the bias does not disappear asymptotically but does not explain why. The set of equations in the middle of the slide exactly matches the picture I have included in my question. This is for a fixed $\lambda$. Since $\lambda\rightarrow 0$ as $n\rightarrow\infty$, the equations imply $\mathbb{E}|\hat\beta_j-\beta_j|\rightarrow 0$. $\endgroup$ Aug 9, 2021 at 17:47
  • $\begingroup$ I don't know the exact argument the authors are using to derive the bias. It isn't as simple as pushing $ n \to \infty$. There are two competing sequences: $n \to \infty$ and $\lambda \to 0$. The $\lambda$ must approach 0 at a rate faster than $n \to \infty$. Otherwise, the parametric $\sqrt{n}$ convergence-rate of $\hat{\beta}_j$ will be the primary driver of convergence to a biased value, $\beta^*_j$. $\endgroup$
    – Eli
    Aug 9, 2021 at 19:38
  • $\begingroup$ Basically, $\lambda$ has to do its' job of selecting the non-zero coefficients quickly then get out of the way before the parameter estimates converge to something. $\endgroup$
    – Eli
    Aug 9, 2021 at 19:41
  • $\begingroup$ Could you explain a bit more on why the two sequences are competing? As I see it, it is enough that $\lambda\rightarrow 0$ (however slowly). That should shrink the gap visible in the graph towards zero. In a sense, I think I am saying that LASSO converges to OLS. But I do not have a proof (or even a sketch of one) for how, say, leave-one-out cross-validation delivers such a $\lambda$; it may not, even though I think it would. $\endgroup$ Aug 9, 2021 at 19:41
  • $\begingroup$ That expectation on slide two is asymptotic. If $\lambda$ is going to 0 slower than $n \to infity$, that bias will never go away. It doesn't matter if $\lambda$ goes to 0 if it is overwhelmed by $n$; relative to any $n$ it will still be cause bias. $\endgroup$
    – Eli
    Aug 9, 2021 at 19:49
0
$\begingroup$

Some literature describes the unbiased nature of SCAD as "nearly unbiased" or "approximately unbiased." However, there is a difference between asymptotic unbiasedness and unbiasedness. The unbiasedness is the small-sample property of OLS estimation, while the asymptotic unbiasedness is the large-sample property of OLS estimation, that is, when the sample size $N$ tends to infinity, the parameter estimator tends to the population value. In fact, as sample size $N$ tends to infinity, the parameter estimates of both LASSO and SCAD satisfy asymptotic unbiasedness according to your simulation codes. So maybe it is more accurate to say that SCAD is estimated to be unbiased when the estimated coefficient is large than $\gamma\lambda$.

$\endgroup$
4
  • $\begingroup$ Thanks. This is roughly how I would think, but I am not sure I am right about this. I am looking for some reference to confirm or disconfirm this. $\endgroup$ Mar 13, 2022 at 13:59
  • $\begingroup$ I've been having trouble understanding the asymptotic unbiasedness of SCAD lately, but testing your R simulation code has thrown light on my understanding of the unbiasedness of SCAD. The common data fitting is usually a small sample, so if LASSO or SCAD is used for data fitting, perhaps the difference between the two methods should be unbiasedness, not asymptotic unbiasedness. I also read some original literature on SCAD, but unfortunately I haven't found any articles to clarify this point. If you find it, please share it. Thank you. $\endgroup$ Mar 13, 2022 at 14:17
  • $\begingroup$ Sure, I will share if I find anything relevant. Javanmard & Montanari "Confidence intervals and hypothesis testing for high-dimensional regression" (2014) is a relevant source but highly technical... $\endgroup$ Mar 13, 2022 at 14:23
  • $\begingroup$ Thanks. I am also not skilled at statistics and mathematics, and maybe it is also hard for me to understand... $\endgroup$ Mar 13, 2022 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.