Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have the following data, which can be modeled by exponential distribution
Time 0-20 20-40 40-60 60-90 90-120 120-inf
Frequency 41 19 16 13 9 2
In order to test if the data follow exponential distribution i will use chi-squared test-statistic. But for this I also need to compute lambda ($MLE = \frac{1}{\bar X}$).
So my question is: how should we choose the midpoint of the interval, if the last interval is from 120 to infinity?
I would not use the midpoint for any of those intervals (expect perhaps as an initial guess for some iterative procedure).
If the data were really from an exponential distribution, the values within each bin should be right skew; the mean would be expected to be left of the average of the bin boundaries.
Note that the equation $\hat{\lambda}=\frac{1}{\bar{X}}$ is suitable if you have all the data. With binned data you need to maximize the likelihood for a binned (i.e. interval-censored) exponential.
[The contribution to log-likelihood of the $n_i$ observations in bin $i$ -- those between $l_i$ and $u_i$ -- is $n_i \log(F(l_i)-F(u_i))$ (where the two terms in $F$ are functions of the parameter(s) of the distribution).]
Because of the lack of memory property of the exponential, if you have a good approximation for the mean of the exponential you also have a good approximation of the amount by which the mean of the distribution above some value $x_0$ exceeds $x_0$.
So (assuming you don't directly maximize the likelihood* on the interval censored data as I suggested), you could begin with some approximate estimate of the mean ($m^{(0)}$ say) and use $120+m^{(0)}$ as a "centre" of the upper tail.
This might then be used to get a better estimate of the parameter (and hence of the mean) and so obtain an improved estimate of the conditional mean in each bin including the top one. [If you want such an approach I would perhaps lean toward doing EM directly.]
Several simple estimates of the mean can be obtained quickly. For example, since 41% of the values occur below 20, $\exp(-\frac{20}{\hat{\lambda}^{(0)}})=1-0.41$ which corresponds to an estimate of the mean close to $38$. Alternatively, one can get a quick eyeball estimate of the median (something less than 30, perhaps about 28), so the mean should be somewhere near $28/\log(2)$, or around $40$.
Either of these would be reasonable to use as an initial guess at how far above 120 to place an estimate for the conditional mean for the last bin.
* An alternative to maximizing the likelihood would be to minimize the chi-square statistic; the same adjustment to d.f. would be used in that instance. The chi-square statistic is relatively easy to calculate, and pretty simple to optimize for a single parameter:
From a theoretical standpoint, the likelihood of the sample that you obtained would be written as $$\mathcal L(\lambda \mid \boldsymbol x) = \prod_{j=1}^m (e^{-\lambda x_{j-1}} - e^{-\lambda x_{j}})^{n_j},$$ where $(x_0, x_1, \ldots, x_m)$ are the bin boundaries (assuming that each bin represents the probability of observing $x_{j-1} < X \le x_j$), and $n_j$ is the number of observations in bin $j$. Here, you have $m = 6$ bins, with $(x_0, x_1, \ldots, x_m) = (0, 20, 40, 60, 90, 120, \infty)$, and $(n_1, \ldots, n_m) = (41, 19, 16, 13, 9, 2)$. In general, maximizing the log-likelihood of this expression will need a numerical approach. Using Mathematica, I obtained the derivative of the log-likelihood as $$\frac{\partial \ell}{\partial \lambda} = \frac{760}{\sinh 10 \lambda +\sinh 20 \lambda} + 1090 \coth 15 \lambda - 3940.$$ This yields the numeric solution $$\hat\lambda \approx 0.025562426096803193.$$
If you are interested in a closed form, simple estimate, the UWSE (Unique Weight Space Estimator) can be helpful. In particular, if $ \ \hat{w_{[0,20]}}\ $ is the relative frequency of observations in the interval $ \ [0,20] \ $, then:
$$ \ \hat{\lambda_{UWSE}} = -\frac{ln(1-\hat{w_{[0,20]}})}{20} \ $$
In this case, $ \ \hat{w_{[0,20]}} = 0.41\ $ , and hence,
$$ \ \hat{\lambda_{UWSE}} = 0.02638164 \ $$
Though, all that can be said of the UWSE is that it is a consistent estimate. Here is a link to the full explanation of the estimator:https://paradsp.wordpress.com/ - scroll all the way to the bottom.
