Optimization (minimization) algorithm - hitting on boundaries

Question

I'm struggling with a challenging optimization problem with real-world experimental data. Simply put, it's about fitting a exponential decay model to a curve (decay). Essentially, I am trying to predict 3 parameters from the curve. The range of expected values for each parameter is well known, so I have set hard boundaries (also to avoid mathematical errors like divide by 0).

If you imagine I have a problem set of hundredths of decays, fitting all of them and making a histogram of each predicted parameter should give somewhat of a normal distribution (roughly) for each parameter. In reality, usually at least two of the parameters are predicted to be at the boundary in 90% of the cases:

This happens wth any method I try, whether non-linear least squares or some of the gradient or gradient-free minimizers.

Can anyone help me out with a professional explanation why this is occurring?

Answer 1

I don't think anything is necessarily going wrong here. A simplified version of your question might be:

I have a biased coin, for which I don't know the probability of flipping a heads. I do know that the probability is uniformly distributed within 40% and 60%, and I can flip the coin once before trying to guess at P(heads).

Obviously, in this case, the MLE/MAP guess will always be exactly 0.4 if my flip lands tails, and 0.6 if it lands heads. I have so little data that I must use the known hard bounds of my prior to constrain my estimate.

In your case, it seems like you do have a stronger than uniform prior, so it seems worth incorporating it in your fitting process. Second, you might be interested in the posterior mean of your paramter, rather than the MLE or MAP. In the simplified coin case, the posterior mean would end up being 49.33% or 50.67% which makes sense, since a single flip doesn't give us that much information.

I'd also watch out for outliers and heteroskedasticity, especially since you mention something which exponentially decays.