I know that, under the usual regularity conditions, the MLE converges to the true parameter values as the sample size gets large. And the scaled MLE tends to being normally distributed. However, in a number of real world cases, finding the global maximum is difficult due to the presence of local minima in the likelihood. Intuitively, I would expect that the number and depth of the local minima would decrease as the sample size increases, leading to a unimodal likelihood. But is this true? Has it been proven?
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ My intuition says no: I believe it is possible to construct a parametric distribution from which one can take infinitely many samples but there exist two distinct vector parameters for which the likelihood is maximized. $\endgroup$– heropupSep 9, 2014 at 17:03
-
$\begingroup$ "the MLE converges to the true parameter values" assumes that you have an algorithm that correctly locates the global maximum. It means that the sequence of values that achieve global maximas converges to the true parameter values, as sample size increases. $\endgroup$– KhashaaDec 25, 2014 at 2:54
-
1$\begingroup$ @heropup That possibility is ruled out by the identification condition which is part and parcel of the regularity conditions. $\endgroup$– KhashaaDec 25, 2014 at 3:04
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Regarding "I would expect that the number and depth of the local minima would decrease as the sample size increases", this is not true in general.
For example, let $X_1,\dots,X_n$ be a random sample from the $k$-component mixture $$ w_1\cdot\mathrm{N}(\mu_1,\sigma_1^2) + \dots + w_k\cdot\mathrm{N}(\mu_k,\sigma_k^2) \, , $$ in which $w_i\geq 0$ and $\sum_{i=1}^k w_i=1$. Define $\theta_i=(w_i,\mu_i,\sigma_i^2)$, for $i=1\dots,k$, and $\theta=(\theta_1,\dots,\theta_k)$, and let $x=(x_1,\dots,x_n)$. The likelihood function is $$ L_x(\theta) = \prod_{i=1}^n \sum_{j=1}^k w_j\cdot\frac{1}{\sqrt{2\pi}\sigma_j} e^{-(x_i-\mu_j)^2/2\sigma_j^2} \, . $$ Since for any permutation $\tau:\{1,\dots,k\}\xrightarrow{\rm 1:1}\{1,\dots,k\}$ we have $$ L_x(\theta_1,\dots,\theta_k) = L_x(\theta_{\tau(1)},\dots,\theta_{\tau(k)}) \, , $$ for this model the likelihood has at least $k!$ symmetric modes, no matter how large the sample size $n$ is.
-
5$\begingroup$ This seems a little beside the point because it's really reflecting the action of a symmetry group on the parameter space leading to an identifiability problem. That is, the space of distributions itself may have exactly one global extremum but the set of parametric representations of that extremum has $k!$ extrema merely because it has a $k!$-fold redundancy. Another simple example of this phenomenon would be estimating the center of a circular Normal distribution: the angle would be uncertain by an integral multiple of $2\pi$, giving infinitely many extrema. $\endgroup$– whuber ♦Sep 9, 2014 at 17:47
-
1$\begingroup$ Maybe bringing a nonidentifiable model is a little unfair, but I think it answers the OP regarding "I would expect that the number and depth of the local minima would decrease as the sample size increases". Well, at least in irregular cases like this it doesn't. $\endgroup$– ZenSep 9, 2014 at 17:58
-
$\begingroup$ Your comment also seems to suggest that consideration of a suitable reparameterization of the model (with the introduction of special constraints, for example, to make $\text{parameter value}\mapsto\text{sampling distribution}$ injective, eliminating the "redudancy") would render void the nonidentifiability issue, but the literature in topics like (lack of) label switching in Gibbs sampling of mixture models indicates that this question is much harder. Sorry if I'm reading what you haven't written! $\endgroup$– ZenSep 9, 2014 at 18:23
-
3$\begingroup$ No problem--those are good ideas regardless. I always appreciate your thoughtful and penetrating posts. It would be nice in this case, though, to see an example that went at the issue in a more fundamental way. (Incidentally, I think a good way to view the situation you describe is that the parameter space is a covering space for the space of distributions.) $\endgroup$– whuber ♦Sep 9, 2014 at 19:23
-
3$\begingroup$ The question actually came up in a discussion of problems estimating neural networks. In that situation, identifiability is a problem, since any permutation of the interior nodes gives the same result. But the loss function tends to have local minima that produce different solutions as well. I was wondering if the optimization problem becomes less ambiguous as the sample size gets large. $\endgroup$– PlacidiaSep 9, 2014 at 21:55