1
$\begingroup$

Short version

Can bootstrap be used to find disconnected confidence regions when MLE is not unique?

Long version Let $\theta$ be a parameter and $P_\theta=\mathrm{Normal}(\theta, 1)$ be a distribution. In the frequentist approach to statistical inference one has access to the sample $X_1, \dotsc, X_n\sim P_\theta$ and can construct the maximum likelihood estimator $\hat\theta(X_1, \dotsc, X_n) = \frac{1}{n}\sum_i X_i$ as well as construct the 95% confidence interval $\mathrm{CI}(X_1, \dotsc, X_n)$ around $\hat \theta$ in various manners (in this case an analytic formula is available, but one could also use likelihood profile, Fisher information matrix, or bootstrap).

My understanding is that:

  • We work with an identifiable model (for $\theta_1\neq \theta_2$ we have $P_{\theta_1}\neq P_{\theta_2}$), so for large $n$ we will find (approximately) unique $P_\theta$ and from this $\theta$ in turn.
  • The confidence intervals based on any of the above methods have their usual meaning, i.e., if we repeat the procedure of sampling the data from $P_\theta$ and construct the confidence interval for each sample, then 95% of them will cover the true value $\theta$.

However, in more complex situations (especially when the model is non-identifiable) a maximum likelihood solution may not be unique. For example, consider $P_\theta=\mathrm{Normal}(\theta^2, 1)$ with two maximum likelihood estimates, $\hat \theta_{\pm}(X_1, \dotsc, X_n) = \pm\sqrt{\frac 1n\sum X_i}$.

In this case I can still define 95% confidence regions (which often will be disconnected in this case, consisting of two intervals) for parameter $\theta$ adjusting the analytical formulas.

However, I do not know how to construct the confidence regions when (a) analytical formulae are not available or (b) I do not even know how many maximum likelihood solutions exist.

Could you recommend me some references on finding confidence regions with non-unique maximum likelihood estimates? I do not know whether methods such as likelihood profile, Fisher information matrix, or (most importantly for me) bootstrap would still work and retain the usual meaning of confidence regions.

I know that in Bayesian statistics identifiability poses different kinds of issues (as label switching and how to understand the multimodality in the posterior), but I would like to learn how this problem can be tackled from the frequentist perspective.

Improve this question
$\endgroup$
5
  • $\begingroup$ It may depend on how exactly non-identifiability plays out in the problem you are interested in. Particularly, is the algorithm to find the estimator constructed in such a way that it won't systematically prefer one parameter to others out of the "non-identifiability set"? If it doesn't, a large enough number of bootstrap replications should actually do the trick. $\endgroup$
    – Christian Hennig
    Commented Apr 5, 2023 at 10:36
  • $\begingroup$ Are you interested in this in general, or rather for a specific situation? If it is the latter, it would be helpful to explain that situation, as what exactly can be done will probably depend on it. $\endgroup$
    – Christian Hennig
    Commented Apr 5, 2023 at 10:58
  • $\begingroup$ Also, whether parameters are not identifiable or whether ML-solutions are not unique is not necessarily the same problem. There are situations in which parameters are identifiable, but for finite $n$ it can happen that ML-solutions are not unique. The issue of not identifiable parameters is sometimes analytically accessible even if it is not possible to be sure how to find all ML-solutions. $\endgroup$
    – Christian Hennig
    Commented Apr 5, 2023 at 11:01
  • $\begingroup$ Thank you very much for your comments and many apologies for the delay in the response! I am interested in general methodology here – I often see that a model is proposed (without any study on identifiability), fitted using gradient ascent (sometimes with several random restarts), and the found parameters (maybe even MLE) are interpreted in light of the preexisting knowledge on the subject, which is used to construct the argument that the proposed model is reasonable. I wanted to know how this point-like estimate can be replaced with something more robust... $\endgroup$
    – Paweł Czyż
    Commented May 26, 2023 at 8:28
  • $\begingroup$ ... as confidence regions. Also, you are right – even if in the large $N$ limit the MLE becomes unique with high probability, for small sample sizes this does not to be the case. While bootstrap sounds interesting, it's performance (even when MLE is unique for any sample size, like the normal distribution) can depend on the number of samples available. I wonder if there exists a theory or good computational study discussing bootstrap in the cases with multiple MLE estimates (or non-identifiable models). $\endgroup$
    – Paweł Czyż
    Commented May 26, 2023 at 8:31

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.