6
$\begingroup$

Let $X_1, X_2,...X_k$ be random independent variables, each $X_i$ drawn from a Geometric distribution $\mathcal{G}(p_i)$, and let its convolution, or sum, be $Y = \sum_{i=1}^k X_i$.

The likelihood function of each $X_i$ with respect to $p_i$ is unimodal (the $r$ parameter is known). But what can we say about the likelihood of the sum (with respect to $p_1,...,p_k$)? $$ \mathcal{L}(\boldsymbol{p} = p_1,...p_k; Y) $$

The context of my problem is the following: I have an EM algorithm to estimate the parameters $\boldsymbol{p} = p_1,p_2,...,p_k$ (given a set of observations $Y_1,...,Y_n$), and I would like to know whether I can say that this is the unique global maximum or whether it might be a local optimum.

Are there some distributions, or family of distributions, for which we have theoretical results?*

I'm looking either for a direct answer or a pointer to some references.

*Appart from the trivial cases such as sums of i.i.d Poissons, sums of Geometrics with same $p$, etc which give known distributions.

$\endgroup$
  • 1
    $\begingroup$ This question makes no sense for single distributions, because your use of "unimodal" necessarily refers to the likelihood as a function of the $p_i$. This makes me wonder whether you might be using "likelihood function" in some nonstandard sense. Maybe you are using it as a synonym for "probability function"? Could you clarify this? $\endgroup$ – whuber May 26 '17 at 14:01
  • $\begingroup$ @whuber, I'm not sure I understand your comment, but I added some details to compensate my lack of (understanding and) proper vocabulary. $\endgroup$ – alberto May 26 '17 at 14:14
  • $\begingroup$ @alberto are you assuming that the $X_i$ are independent? $\endgroup$ – jld May 26 '17 at 14:23
  • $\begingroup$ @Chaconne yes they are! $\endgroup$ – alberto May 26 '17 at 14:24
  • 1
    $\begingroup$ (1) What does the "$r$" parameter represent? AFAIK, a geometric distribution has only one parameter (see en.wikipedia.org/wiki/Geometric_distribution for instance). (2) The sum of IID geometric distributions is a Negative Binomial. Is this perhaps what you are referring to as a "geometric distribution"? $\endgroup$ – whuber May 26 '17 at 14:33
5
+100
$\begingroup$

The short answer is no, it is not a unique global maximum, but there is a longer answer as well.

Consider the sum of two Geometric variates with parameters $p_1 = 0.1,\space p_2 = 0.9$. If you have access to the individual variates, all is well, but if you only have access to the sum, it should be intuitively clear that you will not be able to distinguish the two cases $p_1 =0.1,\space p_2 = 0.9$ and $p_1 = 0.9,\space p_2 = 0.1$. More generally, you will not be able to distinguish $p_1 = a,\space p_2 = b$ from $p_1 = b,\space p_2 = a$, because the distribution of the sum is the same either way. Because of this, depending on your starting values, your algorithm could converge to either (for example) $\hat{p}_1 = 0.134, \space \hat{p}_2 = 0.802$ or $\hat{p}_1 = 0.802, \space \hat{p}_2 = 0.134$, and the associated values of the likelihood function will be the same. This is because the distribution of the sum is independent of the arrangement of the distributions of the random variates across the indices of the random variates being summed. This will always be the case if the components are independent.

In another context - estimation of mixture models - this is related to the "label-switching" problem.

If, on the other hand, you don't care about $p_1,\space p_2,\dots,p_k$ as they relate to the specific unobserved random variables, then things are different. You just want a collection of $k$ estimates, and don't care which is associated with index 1, index 2, etc. You can achieve this by generating your estimates, then ordering them from smallest to largest, for example (there are plenty of other ways of doing it), so $p_1^*$ is your estimate of the smallest $p_i$, etc. In this case, the ordered estimates do form a unique global maximum of the likelihood function - assuming at least $k$ data points - subject to the constraint. But note that you achieve this by, in effect, imposing a constraint on the estimation procedure that forces the ordering $p_1 \leq p_2 \leq \dots \leq p_k$ on the estimates themselves.

$\endgroup$
  • 1
    $\begingroup$ Thank you so much @jbowman ! I'm aware that there will be k! symmetries. But do you mean then that, imposing one of the symmetries, it is unimodal? $\endgroup$ – alberto Jun 10 '17 at 20:59
  • 1
    $\begingroup$ Unimodal over the constrained set, assuming all the usual regularity conditions apply over that set and that your parameters are identifiable (in the case of your example, this is where the "assuming at least $k$ data points" comes into play.) But meeting identifiability and regularity conditions is not always clear - imagine that in all your samples, $X_1 = 0$ - I suspect, writing from intuition, this would violate the conditions. Maybe not, I'll have to think about that. $\endgroup$ – jbowman Jun 11 '17 at 13:49
4
$\begingroup$

jbowman provided an answer regarding the label-switching problem. But putting that aside it made me wonder about the proof for the unique global maximum given the constraint of $p_1 < p_2 < etc$. How can we be sure that the set of p-values for the maximum $\mathcal{L}$ is unique?

I could get an answer in the case of a single measurement $Y$. But still wonder about multiple measurements of $Y$.


Consider a particular set $(X_1, X_2, X_3 ...)$ to obtain the sum $Y=\sum X_i$

Then for each individual variable $X_i$ the maximum likelihood of the associated $p_i$ is $\hat{p_i} = \tfrac{1}{X_i}$ with value $\mathcal{L}(\hat{p_i} \vert X_i) = (1-\tfrac{1}{X_i})^{X_i-1}\tfrac{1}{X_i}$

And for the total set of variables $X_i$ the likelihood (if for each $X_i$ we choose the maximum $p_i$) is $\mathcal{L}(\hat{p_1}, \hat{p_2}, \hat{p_3}, etc. \vert X_1,X_2,X_3,etc.) = \prod \mathcal{L}(\hat{p_i} \vert X_i) = (1-\tfrac{1}{X_1})^{X_1-1}\tfrac{1}{X_1} (1-\tfrac{1}{X_2})^{X_2-1}\tfrac{1}{X_2} (1-\tfrac{1}{X_3})^{X_3-1}\tfrac{1}{X_3} etc$

For a given set $X$ we can analyze two separate terms in the $\mathcal{L}$-function. And consider whether the likelihood increases if we would choose two different terms with the same sum. This relates to evaluating the maximum of the related terms parameterized by $m$ and $x$ (with $m$ the midpoint of the two variables $X_i$ and $x$ the difference of the variables from the midpoint):

$\frac{1}{m+x}\frac{1}{m-x}(1-\frac{1}{m+x})^{m+x-1}(1-\frac{1}{m-x})^{m-x-1}$

whose derivative for $x$ is equal to

$\frac{(1-\frac{1}{m-x})^{(m-x)}(1-\frac{1}{m+x})^{(m+x)}\left( log(1-\frac{1}{m+x}) log(1-\frac{1}{m-x}) \right) }{(m-x-1)(m+x-1)}$

from which we can conclude that the maximum is present $x=0$ and that the closer the variables in the set $X$, or the more uniform, the higher the likelihood.

And thus there is a unique set of $p_i$ (the most homogeneous set $X$ of variables $X_i$ such that we can't select any pair from $X$ for which we could decrease the value of $x$ according to the previous discussed pattern) for the maximum likelihood given a single $Y$ and a given number size of the set of variables $X$.

$\endgroup$
  • 1
    $\begingroup$ I did not take into account that there are more ways in which a sum can be generated for an inhomogeneous sample. Say you measure $Y=9$ and assume this a sum of three variables $X_1, X_2, X_3$ then there is only one way in which you encounter $3, 3, 3$ and there are multiple ways for $3, 2, 4$ which means that the set of p-values associated with the non-uniform set of $X$ might actually have a higher likelihood. I wonder if there can be an easy example which shows that for some situation we can have $\mathcal{L}(X_i,X_i)$=$\mathcal{L}(X_j,X_k)$, because of this multiplicity. $\endgroup$ – Sextus Empiricus Jun 13 '17 at 23:47
  • 1
    $\begingroup$ This is an interesting question you raise - would you like (or perhaps you already have) to post it as a question in its own right? $\endgroup$ – jbowman Jun 28 '17 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.