1
$\begingroup$

I am using PROC FMM in SAS, in attempt to use hurdle models on a data set with many zeros. There is one response variable and it's continuous, there are ~90 predictors (continuous - but contain many zeros) and roughly 160 000 records. When I tried running this procedure, I was told

"WARNING: Dual Quasi-Newton optimization cannot be completed. NOTE: The Dual Quasi-Newton optimization technique needs more than 200 iterations or 2000 function calls. Error: No final model fitted because no 'best' model can be determined."

I then tried this same process with fewer variables and the optimization process worked (I think). What possible reasons could there be for this? Additionally, does anyone recommend other prediction/discrimination techniques to use in heavy zero data sets?

$\endgroup$
  • 1
    $\begingroup$ Although this question is asked in the context of SAS, this seems to be a statistical issue. IMO, this can be considered on topic here. $\endgroup$ – gung Oct 29 '15 at 19:15
  • $\begingroup$ Sometimes such behavior can be due to just one or a small number of variables. Have you tried to narrow down the scope of the problem by exploring the effects of adding and removing variables? $\endgroup$ – whuber Oct 31 '15 at 12:46
  • $\begingroup$ Paul Allison's SAS proceedings paper on Convergence Failures in Logistic Regression contains insights that are generalizable to many other maximum likelihood rooted SAS Procs as well as other software. www2.sas.com/proceedings/forum2008/360-2008.pdf $\endgroup$ – DJohnson Oct 31 '15 at 13:27
1
$\begingroup$

It's the curse of dimensionality. Luckily, SAS can actually throw an error. With any type of non-linear optimization, you must realize more sophisticated models lead to highly irregular likelihoods that toss general optimizers all over the place.

Why not use a quasipoisson model?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.