2
$\begingroup$

I am working with a non-linear model that uses seven independent variables to estimate a Bernoulli probability.

To estimate the parameters of the model, I am optimizing a likelihood function using the optim function in R. I have tried several of the optimizing algorithms available in the optim function.

The optimizing algorithm is frequently running into saddle points. Mathematically, I understand how increasing the dimensions increases the number of saddle points the optimizing algorithm can run into. I also understand how my non-linear model makes this a non-Convex optimization problem.

What I am trying to obtain is an intuitive understanding of what is it about the nature of the data that could make this model run into a saddle point? Obviously you don't have my data at hand and can't answer specifically, but I want to understand what is it in general that can make a model run into a saddle point for one data set but not another.

$\endgroup$
6
  • $\begingroup$ It's just the nature of the geometry of the data set, it has a lot of saddle points. A quick trick to try and avoid getting stuck at a saddle point is to rerun the optimization with a new starting value. However if your data is littered with saddle points this probably won't help. $\endgroup$ – Wintermute Jan 8 '16 at 15:21
  • $\begingroup$ I suppose what I'm trying to understand is what is it about the data that can cause so many saddle points? Is there some inherent limitation or flaw in the data that causes the parameters to be difficult to estimate? I have a very intuitive understanding on why, for example, multicollinearity causes parameters to be difficult to estimate. I'm hoping to gain something like that here. $\endgroup$ – TrynnaDoStat Jan 8 '16 at 15:27
  • $\begingroup$ Technical, but may help.arxiv.org/abs/1406.2572 $\endgroup$ – Wintermute Jan 8 '16 at 15:42
  • $\begingroup$ It's totally dependent on your likelihood function. Without knowing your likelihood, there's no way anyone can say why the likelihood function is multimodal. In addition, if you're using optim, I'm assuming it's a custom likelihood, which increases the odds of there being a bug in the likelihood function code. $\endgroup$ – Cliff AB Jan 8 '16 at 16:52
  • $\begingroup$ I mention that last bit because many classic log likelihood functions are concave under mild conditions, making optimization much easier. Machine learning researchers seem less concerned with creating convex loss functions than statisticians. I would suggest this is a result of statisticians starting from "given X assumptions of data, what is the best estimator?", where as ML researchers tend to approach it as "I want a really good estimator without worrying too much about the math". But I digress. $\endgroup$ – Cliff AB Jan 8 '16 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.