Consider an arbitrary-ish regression model with the unpenalized likelihood $$ \log \mathcal{L} = \sum_i f\left(y_i\,|\,g(\beta_0 + \beta x_i)\right) $$

with $\beta = \left(\beta_1, \dots, \beta_K\right)$ and $x_i$ being a data vector.

If I instead maximize $$ \log \mathcal{L} = \sum_i f\left(y_i\,|\,g(\beta_0 + \beta x_i)\right) + \lambda P_\alpha(\beta) $$ where $P_\alpha(\beta) = \sum_{k=1}^K \frac{1}{2}(1-\alpha)\beta_k^2 + \alpha | \beta_k |$, will my estimated $\beta$ have the same nice properties for an arbitrary convex $f$ as for a well-studied $f$ like the Bernoulli or Gaussian likelihoods studied in Friedman, Hastie, and Tibshirani (2010)?

Specifically, what if $y$ is censored? That is, $y = \cases{y^* &\text{if}\ y^* < c \\ c &\text{if}\ y^* \geq c}$. Then my likelihood is something like $$ \log \mathcal{L} = \sum_{i\ : y^*_i < c} F^*\left(y^*_i\,|\,g(\beta_0 + \beta x_i)\right)\,f^*\left(y^*_i\,|\,g(\beta_0 + \beta x_i)\right) \\+ \sum_{i\ : y^*_i \geq c} 1 - F^*(c\,|\,g(\beta_0 + \beta x_i)) \\+ \lambda P_\alpha(\beta) $$ Is there any reason why this wouldn't work?

I imagine this is related to the question Do I get the nice asymptotic properties of MLE when I restrict the parameter space?


I'll try to address this question in a general way. For generalized linear models (GLM), it makes complete sense to use elastic-net priors on the parameters as you describe. This is merely to say that the standard elastic-net regularization framework works out-of-the-box for a broader class of models than simply Bernoulli and Gaussian observation models which are simple examples of GLMs, (e.g. Poisson observations correspond to another GLM).

Each GLM has a canonical link and when using this link function, the GLM is log-concave, and the negative log likelihood (NLL) is convex. The L1 and L2 penalties are convex. The sum of these convex penalties with the convex objective (NLL) gives an overall convex objective.

The possible issues arise is if you use a non-standard link (specific non-standard links won't break things) or aren't using a GLM. Basically for an arbitrary regression objective, it may be the case that the NLL part of your objective will no longer be convex (although the penalties will remain convex). Even if the NLL is not convex, the penalties will affect the estimated weights, but guarantees are weaker since you will obtain locally optimal weights.

For specific examples you have, you could check whether your objective function is convex before the addition of penalties. If it is, then the addition of convex penalties usually preserves convexity. If the original objective is not convex, then the messiness is probably coming from that rather than the penalties.

  • 1
    $\begingroup$ "the addition of convex penalties usually preserves convexity" – it always does, the sum of convex functions is convex, and the question specified a convex $f$. I think the question is about whether the solution will be sparse and reasonably regularized in a "nice" way, not whether the optimization problem remains tractable. $\endgroup$ – Dougal Apr 21 '15 at 23:56
  • 1
    $\begingroup$ @Dougal indeed that's what I'm after. But this is a nice start $\endgroup$ – shadowtalker Apr 23 '15 at 18:50
  • $\begingroup$ For a specific case with a convex objective and a mixture of L1 and L2 penalties, my qualitative take would be that the effect of the penalties should be interpretable -- they will respectively have sparsening and shrinkage effects. Especially within the GLM context, you can think of the penalties as priors on the weight distribution. If you needed to verify a specific property for a specific model, perhaps the probabilistic interpretation might be an option for a starting point. $\endgroup$ – Josh Apr 23 '15 at 20:23
  • $\begingroup$ @Josh thanks for clarifying. It seems like "all roads lead to Bayes" or something like that. $\endgroup$ – shadowtalker Apr 29 '15 at 3:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.