1
$\begingroup$

Consider any log likelihood function $f(\theta|x)$ where $x$ is data. I can consider $f(\theta|x)+\lambda||\theta||_1$ where $||\theta||_1$ is the standard $L_1$ norm.

It seems that I cannot apply this sort of argument naively to obtain sparsity. I found in the following article that conditional "independence" is required. I think conditional independence in this case is equivalent to sparsity. https://tibshirani.su.domains/ftp/graph.pdf

(pg 1)"The basic model for continuous data assumes that the observations have a multivariate Gaussian distribution with mean $\mu$ and covariance matrix $\Sigma$. If the $ij$th component of $\Sigma^{-1}$ is zero, then variables $i$ and $j$ are conditionally independent, given the other variables. Thus it makes sense to impose an $L_1$ penalty for the estimation of $\Sigma^{-1}$, to increase sparsity."

I think the paper is used to estimate $\Sigma^{-1}$ which is sparse. Why conditional independence is required to make sense for sparsity by $L_1$? It seems that variables $i$ and $j$ are conditional independent if and only if $ij$th component of $\Sigma^{-1}$ is zero. That is reason why sparsity is expected by conditional independence.

$\endgroup$
7
  • $\begingroup$ no that's a misunderstanding. For their application (with gaussian data), having the matrix elements being zero corresponds to conditional independence. For other applications, eg linear regression, it's not related. $\endgroup$
    – seanv507
    Apr 8 at 15:37
  • $\begingroup$ @seanv507 However, if I remember correctly from ESL, the book does mention how to derive ridge case from independence of parameter $\theta$ in regression case. That independence is the assumption to derive degree of freedom in lasso case. $\endgroup$
    – user45765
    Apr 8 at 16:17
  • 1
    $\begingroup$ Should $f(\theta|x)$ not be $f(x|\theta)$? $\endgroup$ Apr 9 at 7:22
  • $\begingroup$ @RichardHardy They are same thing under maximum likelihood without Bayesian view. They maybe completely different under Bayesian if one uses prior. $\endgroup$
    – user45765
    Apr 9 at 13:19
  • $\begingroup$ I think that from a frequentist perspective, $f(\theta|x)$ is not meaningful, as a parameter does not have a probability density function. $\endgroup$ Apr 9 at 13:27

1 Answer 1

3
$\begingroup$

I found in the following article that conditional "independence" is required.

This condition relates to a different issue which is specific to the application in that article.

It is not a condition that is necessary in order to be able to apply a L1 norm to obtain sparsity.

That idea of independence relates to the model

The basic model for continuous data assumes that the observations have a multivariate Gaussian distribution with mean $\mu$ and covariance matrix $\Sigma$

For this model independence between components $i$ and $j$ is equivalent to zero covariance $\Sigma_{ij}$ (and those covariance terms is what is being estimated in this specific case).

The zero's $\Sigma_{ij}$ are a motivation to use estimation of the $\hat\Sigma_{ij}$ with an L1-norm regularisation. But for different problems, where not the parameters $\Sigma_{ij}$ are being estimated but something else, it will not be necessary and we would expect the sparsity to be present in some other way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.