10
$\begingroup$

Is it possible to apply the usual MLE procedure to the triangle distribution? - I am trying but I seem to be blocked at one step or another in the math by the way the distribution is defined. I am trying to use the fact that I know the number of samples above and below c (without knowing c): these 2 numbers are cn and (1-c)n, if n is the total number of samples. However, that does not seem to help in the derivation. The moment of moments gives an estimator for c without much problem. What is the exact nature of the obstruction to MLE here (if indeed there is one)?

More details:

Let's consider $c$ in $[0,1]$ and the distribution defined on $[0,1]$ by:

$f(x;c) = \frac{2x}{c}$ if x < c
$f(x;c) = \frac{2(1-x)}{(1-c)}$ if c <= x

Let's take an $n$ i.i.d samples $\{x_{i}\}$ from this distribution form the log-likelihood of c given this sample:

$\hat{l}(c | \{x_{i}\}) = \sum_{i=1}^{n}ln(f(x_{i}|c))$

I am then trying to use the fact that given the form of $f$, we know that $cn$ samples will fall below the (unknown) $c$, and $(1-c)n$ will fall above $c$. IMHO, this allows to decompose the summation in the expression of the log-likelihood thus:

$\hat{l}(c | \{x_{i}\}) = \sum_{i=1}^{cn}ln\frac{2 x_{i}}{c} + \sum_{i=1}^{(1-c)n}ln\frac{2(1-x_{i})}{1-c}$

Here, I am unsure how to proceed. MLE will involve taking a derivative w.r.t. $c$ of the log-likelihood, but I have $c$ as the upper bound of the summation, which seems to block that. I could try with another form of the log-likelihood, using indicator functions:

$\hat{l}(c | \{x_{i}\}) = \sum_{i=1}^{n}\{x_{i}<c\}ln\frac{2 x_{i}}{c} + \sum_{i=1}^{n}\{c<=x_{i}\}ln\frac{2(1-x_{i})}{1-c}$

But deriving the indicators doesn't seem easy either, although Dirac deltas could allow to continue (while still having indicators, since we need to derive products).

So, here I am blocked in MLE. Any idea?

$\endgroup$
  • $\begingroup$ If this is for some subject please add the self-study tag. If it isn't, please explain how the problem arises. $\endgroup$ – Glen_b Jul 12 '13 at 0:12
  • $\begingroup$ Thanks for the update; it makes it much easier to say sensible things in answer, since it vastly reduces the scope of cases to deal with. Could you please consider my earlier comment. Either this falls under the self-study tag or it doesn't, in either case I have asked if you would do something. $\endgroup$ – Glen_b Jul 12 '13 at 1:10
  • $\begingroup$ This not for a homework or a class. It arises at my work. We have another estimator from method of moments, but I'm trying to get a deeper understanding of what is going on with MLE here. $\endgroup$ – Frank Jul 12 '13 at 1:54
  • $\begingroup$ Okay; that gives me more leeway. See my updated answer. I will probably make further additions soon $\endgroup$ – Glen_b Jul 12 '13 at 2:12
  • $\begingroup$ Added references/links $\endgroup$ – Glen_b Jul 12 '13 at 2:30
10
$\begingroup$

Is it possible to apply the usual MLE procedure to the triangle distribution?

Certainly! Though there are some oddities to deal with, it's possible to compute MLEs in this case.

However, if by 'the usual procedure' you mean 'take derivatives of the log-likelihood and set it equal to zero', then maybe not.

What is the exact nature of the obstruction to MLE here (if indeed there is one)?

Have you tried drawing the likelihood?

--

Followup after clarification of question:

The question about drawing the likelihood was not idle commentary, but central to the issue.

MLE will involve taking a derivative

No. MLE involves finding the argmax of a function. That only involves finding the zeros of a derivative under certain conditions... which don't hold here. At best, if you manage to do that you'll identify a few local minima.

As my earlier question suggested, look at the likelihood.

Here's a sample, $y$ of 10 observations from a triangular distribution on (0,1):

0.5067705 0.2345473 0.4121822 0.3780912 0.3085981 0.3867052 0.4177924
0.5009028 0.8420312 0.2588613

Here's the likelihood and log-likelihood functions for $c$ on that data: likelihood for peak of triangular

log-likelihood for peak of triangular

The grey lines mark the data values (I should probably have generated a new sample to get better separation of the values). The black dots mark the likelihood / log-likelihood of those values.

Here's a zoom in near the maximum of the likelihood, to see more detail:

Detail of likelihood

As you can see from the likelihood, at many of the order statistics, the likelihood function has sharp 'corners' - points where the derivative doesn't exist (which is no surprise - the original pdf has a corner and we're taking a product of pdfs). This (that there are cusps at order statistics) is the case with the triangular distribution, and the maximum always occurs at one of the order statistics. (That cusps occur at order statistics isn't unique to the triangular distributions; for example the Laplace density has a corner and as a result the likelihood for its center has one at each order statistic.)

As it happens in my sample, the maximum occurs as the fourth order statistic, 0.3780912

So to find the MLE of $c$ on (0,1), just find the likelihood at each observation. The one with the biggest likelihood is the MLE of $c$.

A useful reference is chapter 1 of "Beyond Beta" by Johan van Dorp and Samuel Kotz. As it happens, Chapter 1 is a free 'sample' chapter for the book - you can download it here.

There's a lovely little paper by Eddie Oliver on this issue with the triangular distribution, I think in American Statistician (which makes basically the same points; I think it was in a Teacher's Corner). If I can manage to locate it I'll give it as a reference.

Edit: here it is:

E. H. Oliver (1972), A Maximum Likelihood Oddity,
The American Statistician, Vol 26, Issue 3, June, p43-44

(publisher link)

If you can easily get hold of it, it's worth a look, but that Dorp and Kotz chapter covers most of the relevant issues so it's not crucial.


By way of followup on the question in comments - even if you could find some way of 'smoothing off' the corners, you'd still have to deal with the fact that you can get multiple local maxima:

two local max

It might, however, be possible to find estimators that have very good properties (better than method of moments), which you can write down easily. But ML on the triangular on (0,1) is a few lines of code.

If it's a matter of huge amounts of data, that, too, can be dealt with, but would be another question, I think. For example, not every data point can be a maximum, which reduces the work, and there are some other savings that can be made.

$\endgroup$
  • $\begingroup$ Thanks - I'll try to post my failed attempt, showing what distribution I'm exactly talking about and where I think I am blocked. $\endgroup$ – Frank Jul 12 '13 at 0:25
  • $\begingroup$ Thanks for the detailed explanation! I had another idea though: suppose I could find a family of functions that converges to the triangle distribution, but would not be piecewise - could I use that to derive a MLE analytically, then take the limit and assume I would have a MLE of the triangle distribution itself? $\endgroup$ – Frank Jul 12 '13 at 4:40
  • $\begingroup$ Possibly - I think that might depend on the particular limit process you use ... and you'll likely still end up with several local maxima so it probably only saves you evaluating the likelihood near the extreme order statistics anyway -- but even if it worked, why would you even try to do something so complicated? What's wrong with ML on the triangular distribution? It's really quite simple to do in practice. $\endgroup$ – Glen_b Jul 12 '13 at 4:53
  • 2
    $\begingroup$ I must say, this MLE for c based on order statistics is pretty nice, although the derivation in the chapter above takes some work (not too hard though) - nice illustration that the essence of MLE is in the argmax (of course!), rather than the derivative (as you pointed out, and I fully agree, it occurred to me to work upstream of the "usual" derivative step (i.e. just worry about maximizing, by whatever means), but I didn't pursue). $\endgroup$ – Frank Jul 12 '13 at 6:07
  • 1
    $\begingroup$ @Frank: An additional reference is Huang and Shen (2007) More maximum likelihood oddities, Journal of Statistical Planning and Inference, Volume 137, Issue 7, July, pp 2151-2155. Glen: By order statistics, do you just mean the ordered values $x_{i}$? $\endgroup$ – COOLSerdash Jul 12 '13 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.