# Why lasso cannot be arbitrarily applied?

Consider any log likelihood function $$f(\theta|x)$$ where $$x$$ is data. I can consider $$f(\theta|x)+\lambda||\theta||_1$$ where $$||\theta||_1$$ is the standard $$L_1$$ norm.

It seems that I cannot apply this sort of argument naively to obtain sparsity. I found in the following article that conditional "independence" is required. I think conditional independence in this case is equivalent to sparsity. https://tibshirani.su.domains/ftp/graph.pdf

(pg 1)"The basic model for continuous data assumes that the observations have a multivariate Gaussian distribution with mean $$\mu$$ and covariance matrix $$\Sigma$$. If the $$ij$$th component of $$\Sigma^{-1}$$ is zero, then variables $$i$$ and $$j$$ are conditionally independent, given the other variables. Thus it makes sense to impose an $$L_1$$ penalty for the estimation of $$\Sigma^{-1}$$, to increase sparsity."

I think the paper is used to estimate $$\Sigma^{-1}$$ which is sparse. Why conditional independence is required to make sense for sparsity by $$L_1$$? It seems that variables $$i$$ and $$j$$ are conditional independent if and only if $$ij$$th component of $$\Sigma^{-1}$$ is zero. That is reason why sparsity is expected by conditional independence.

• no that's a misunderstanding. For their application (with gaussian data), having the matrix elements being zero corresponds to conditional independence. For other applications, eg linear regression, it's not related. Apr 8 at 15:37
• @seanv507 However, if I remember correctly from ESL, the book does mention how to derive ridge case from independence of parameter $\theta$ in regression case. That independence is the assumption to derive degree of freedom in lasso case. Apr 8 at 16:17
• Should $f(\theta|x)$ not be $f(x|\theta)$? Apr 9 at 7:22
• @RichardHardy They are same thing under maximum likelihood without Bayesian view. They maybe completely different under Bayesian if one uses prior. Apr 9 at 13:19
• I think that from a frequentist perspective, $f(\theta|x)$ is not meaningful, as a parameter does not have a probability density function. Apr 9 at 13:27

I found in the following article that conditional "independence" is required.

This condition relates to a different issue which is specific to the application in that article.

It is not a condition that is necessary in order to be able to apply a L1 norm to obtain sparsity.

That idea of independence relates to the model

The basic model for continuous data assumes that the observations have a multivariate Gaussian distribution with mean $$\mu$$ and covariance matrix $$\Sigma$$

For this model independence between components $$i$$ and $$j$$ is equivalent to zero covariance $$\Sigma_{ij}$$ (and those covariance terms is what is being estimated in this specific case).

The zero's $$\Sigma_{ij}$$ are a motivation to use estimation of the $$\hat\Sigma_{ij}$$ with an L1-norm regularisation. But for different problems, where not the parameters $$\Sigma_{ij}$$ are being estimated but something else, it will not be necessary and we would expect the sparsity to be present in some other way.