7
$\begingroup$

I have the following data, which can be modeled by exponential distribution

Time        0-20  20-40    40-60  60-90   90-120    120-inf
Frequency   41     19       16      13        9        2

In order to test if the data follow exponential distribution i will use chi-squared test-statistic. But for this I also need to compute lambda ($MLE = \frac{1}{\bar X}$).

So my question is: how should we choose the midpoint of the interval, if the last interval is from 120 to infinity?

$\endgroup$
9
$\begingroup$

I would not use the midpoint for any of those intervals (expect perhaps as an initial guess for some iterative procedure).

If the data were really from an exponential distribution, the values within each bin should be right skew; the mean would be expected to be left of the average of the bin boundaries.

Note that the equation $\hat{\lambda}=\frac{1}{\bar{X}}$ is suitable if you have all the data. With binned data you need to maximize the likelihood for a binned (i.e. interval-censored) exponential.

[The contribution to log-likelihood of the $n_i$ observations in bin $i$ -- those between $l_i$ and $u_i$ -- is $n_i \log(F(l_i)-F(u_i))$ (where the two terms in $F$ are functions of the parameter(s) of the distribution).]

Because of the lack of memory property of the exponential, if you have a good approximation for the mean of the exponential you also have a good approximation of the amount by which the mean of the distribution above some value $x_0$ exceeds $x_0$.

So (assuming you don't directly maximize the likelihood* on the interval censored data as I suggested), you could begin with some approximate estimate of the mean ($m^{(0)}$ say) and use $120+m^{(0)}$ as a "centre" of the upper tail.

This might then be used to get a better estimate of the parameter (and hence of the mean) and so obtain an improved estimate of the conditional mean in each bin including the top one. [If you want such an approach I would perhaps lean toward doing EM directly.]

Several simple estimates of the mean can be obtained quickly. For example, since 41% of the values occur below 20, $\exp(-\frac{20}{\hat{\lambda}^{(0)}})=1-0.41$ which corresponds to an estimate of the mean close to $38$. Alternatively, one can get a quick eyeball estimate of the median (something less than 30, perhaps about 28), so the mean should be somewhere near $28/\log(2)$, or around $40$.

Either of these would be reasonable to use as an initial guess at how far above 120 to place an estimate for the conditional mean for the last bin.

* An alternative to maximizing the likelihood would be to minimize the chi-square statistic; the same adjustment to d.f. would be used in that instance. The chi-square statistic is relatively easy to calculate, and pretty simple to optimize for a single parameter:
enter image description here

$\endgroup$
7
$\begingroup$

From a theoretical standpoint, the likelihood of the sample that you obtained would be written as $$\mathcal L(\lambda \mid \boldsymbol x) = \prod_{j=1}^m (e^{-\lambda x_{j-1}} - e^{-\lambda x_{j}})^{n_j},$$ where $(x_0, x_1, \ldots, x_m)$ are the bin boundaries (assuming that each bin represents the probability of observing $x_{j-1} < X \le x_j$), and $n_j$ is the number of observations in bin $j$. Here, you have $m = 6$ bins, with $(x_0, x_1, \ldots, x_m) = (0, 20, 40, 60, 90, 120, \infty)$, and $(n_1, \ldots, n_m) = (41, 19, 16, 13, 9, 2)$. In general, maximizing the log-likelihood of this expression will need a numerical approach. Using Mathematica, I obtained the derivative of the log-likelihood as $$\frac{\partial \ell}{\partial \lambda} = \frac{760}{\sinh 10 \lambda +\sinh 20 \lambda} + 1090 \coth 15 \lambda - 3940.$$ This yields the numeric solution $$\hat\lambda \approx 0.025562426096803193.$$

$\endgroup$
0
$\begingroup$

If you are interested in a closed form, simple estimate, the UWSE (Unique Weight Space Estimator) can be helpful. In particular, if $ \ \hat{w_{[0,20]}}\ $ is the relative frequency of observations in the interval $ \ [0,20] \ $, then:

$$ \ \hat{\lambda_{UWSE}} = -\frac{ln(1-\hat{w_{[0,20]}})}{20} \ $$

In this case, $ \ \hat{w_{[0,20]}} = 0.41\ $ , and hence,

$$ \ \hat{\lambda_{UWSE}} = 0.02638164 \ $$

Though, all that can be said of the UWSE is that it is a consistent estimate. Here is a link to the full explanation of the estimator:https://paradsp.wordpress.com/ - scroll all the way to the bottom.

$\endgroup$
  • $\begingroup$ This is an interesting idea, but it looks like it wasn't intended to be used in situations where you have far more information than you are actually using. In the present case there are counts of six non-overlapping bins. It would be a shame to ignore five of those counts arbitrarily. $\endgroup$ – whuber Jun 20 '17 at 18:57
  • 1
    $\begingroup$ You are absolutely right. The UWSE was intended to work off of minimal information. It would be interesting to see what other applications arise. In this case I thought it would be beneficial for those who don't want to get into numerical work. @Glen_b explains this above , but it ultimately falls under the category of UWSE - which is more general. $\endgroup$ – CYP450 Jun 20 '17 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.