I understand EM algorithm is often used for missing data/mixture problem. But can it be used to optimize a particular type of likelihood based on jointly fitting variables and transformations of those variables?

I am fitting a logistic regression model: $Y$ is the usual binary dependent variable, $\mathbf{X}'$ is a matrix of independent variables. However, the data $\mathbf{X}' = f_{\alpha}(\mathbf{X})$ are a function of some parameters, $\alpha$. Specifically $X_{.,j}' = \alpha_j^{X_{.,j}}$. We observe only $\mathbf{X}$ and $Y$.

This gives the linear model:

$\mbox{logit} (Y|X) = \beta_0 + \beta_1 \alpha_1 ^{X_1} + \ldots + \beta_k \alpha_k ^{X_k} $

Is there any other method that jointly estimate this model? The one that I can think of is to use a global search. Steps are:

  1. Randomly generate the three alpha parameters within its definition space, fit the model to get coefficient and AUC.
  2. Repeat step 1 for 1000 times.
  3. Choose the alphas that gives the best AUC.

Does this approach make sense? What I am trying to do is to get the optimized parameter. Thank you.

  • $\begingroup$ The first thing you'll need to do is verify whether or not this is a convex log likelihood. $\endgroup$ Sep 18, 2013 at 10:35

1 Answer 1


The EM algorithm is not applied here because there is no expectation step necessary to maximize the likelihood for the logistic model above. If doing this in R, consider using nlm, i.e. straightforward maximum likelihood can jointly estimate the vector of $\beta$ and $\alpha$ in your problem.

loglik <- function(params) {
  dbinom(params[1:k] %*% params[{k+1}:2k]^X, log=TRUE)

If the goal is maximizing the AUC, then you ought to consider an alternate estimation strategy based on ROC regression, though this can be very perilous because it doesn't have nearly the same robust flexibility as maximum likelihood does.

  • $\begingroup$ Thank you for your comment. My goal is to optimize model accuracy which is AUC. If I use nlm function in R, what would be the f function? Since AUC is not a simple function, it requires splitting training data/test data, and modeling to get AUC. Shall I define a customized function to get AUC, then use this customized AUC as input function in nlm? $\endgroup$ May 20, 2013 at 17:13
  • $\begingroup$ Per your second point that my proposed method is not as robust as maximum likelihood, what if I increase the number of repetition and use average of the top x% results? Say, I repeat 1st step for 10000 times so that I got 10000 set of parameter estimates, then I order them by AUC desc, choose the top 1% estimates, (100 estimates in this case) and use the average of these 100 estimates as my final estimate? $\endgroup$ May 20, 2013 at 17:16
  • 1
    $\begingroup$ AUC is not a measure of model accuracy but is a measure of model recall. Some models have perfect recall but lousy classification accuracy. You can estimate your model parameters based on maximizing AUC but expect frustrating convergence issues with almost any initializing parameter sets (even those from ML as starting points). I really can't advocate your ad-hoc method. You ought to consult the Elements of Statistical Learning for some other approaches to this problem. $\endgroup$
    – AdamO
    May 20, 2013 at 17:24
  • $\begingroup$ @AdamO this is incorrect. A high AUC requires a model that can have both high precision and high recall. In more formal terms, "a curve dominates in [AUC] space if and only if it dominates in [Precision/Recall] space". mark.goadrich.com/articles/davisgoadrichcamera2.pdf $\endgroup$ Aug 3, 2014 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.