I have heard of Owen's empirical likelihood, but until recently paid it no heed until I came across it in a paper of interest (Mengersen et al. 2012).

In my efforts to understand it, I have gleaned that the likelihood of the observed data is represented as $$L = \prod_i p_i = \prod_i P(X_i=x) = \prod_i P(X_i \le x) - P(X_i \lt x)$$ , where $\sum_i p_i = 1$ and $p_i > 0$.

However, I have been unable to make the mental leap connecting this representation with how it can be used to make inferences about observations. Perhaps I am too rooted in thinking of a likelihood w.r.t parameters of a model?

Regardless, I have been searching Google Scholar for some paper employing empirical likelihood that would help me internalize the concept... to no avail. Obviously, there is Art Owen's book on Empirical Likelihood, but Google Books leaves out all the yummy bits and I'm still in the slow process of getting an inter-library loan.

In the meantime, can somebody kindly point me to papers and documents that clearly illustrate the premise of empirical likelihood and how it is employed? An illustrative description of EL itself would also be welcome!

  • 2
    $\begingroup$ Econometricians, in particular, have fallen in love with EL. If you're looking for applications, that literature may be one of the better places to look. $\endgroup$ – cardinal Jun 25 '12 at 12:48

I can think of no better place than Owen's book to learn about empirical likelihood.

One practical way to think about $L = L(p_1, \ldots, p_n)$ is as the likelihood for a multinomial distribution on the observed data points $x_1, \ldots, x_n$. The likelihood is thus a function of the probability vector $(p_1, \ldots, p_n)$, the parameter space is really the $n$-dimensional simplex of probability vectors, and the MLE is putting weight $1/n$ on each of the observations (supposing they are all different). The dimension of the parameter space increases with the number of observations.

A central point is that empirical likelihood gives a method for computing confidence intervals by profiling without specifying a parametric model. If the parameter of interest is the mean, $\mu$, then for any probability vector $p = (p_1, \ldots, p_n)$ we have that the mean is
$$\mu(p) = \sum_{i=1}^n x_i p_i,$$ and we can compute the profile likelihood as $$L_{\text{prof}}(\mu) = \max \{ L(p) \mid \mu(p) = \mu \}.$$ Then we can compute confidence intervals of the form $$I_r = \{ \mu \mid L_{\text{prof}}(\mu) \geq r L_{\text{prof}}(\bar{x}) \}$$ with $r \in (0,1)$. Here $\bar{x}$ is the empirical mean and $L_{\text{prof}}(\bar{x}) = n^{-n}$. The intervals $I_r$ should perhaps just be called (profile) likelihood intervals since no statement about coverage is made upfront. With decreasing $r$ the intervals $I_r$ (yes, they are intervals) form a nested, increasing family of confidence intervals. Asymptotic theory or the bootstrap can be used to calibrate $r$ to achieve 95% coverage, say.

Owen's book covers this in detail and provides extensions to more complicated statistical problems and other parameters of interest.

  • 4
    $\begingroup$ (+1) Lacking access to the book, one can always start with the original papers to get the basics of the theory. Like the book, the papers are also quite clearly written. $\endgroup$ – cardinal Jun 25 '12 at 10:01
  • 6
    $\begingroup$ Some links: (1) A. Owen (1988), Empirical likelihood ratio confidence intervals for a single functional, Biometrika, vol. 75, No. 2, pp. 237-249, (2) A. Owen (1990), Empirical likelihood ratio confidence regions, Ann. Statist., vol. 18, no. 1, pp. 90-120 (open access), and (3) A. Owen (1991) Empirical likelihood for linear models, Ann. Statist., vol. 19, no. 4, pp. 1725-1747 (open access). $\endgroup$ – cardinal Jun 25 '12 at 16:41
  • $\begingroup$ @cardinal Fantastic! Should have thought of that myself. $\endgroup$ – Sameer Jun 25 '12 at 19:07
  • $\begingroup$ @NHS Thank you for your explanation! Just to be clear, is $L_{prof}(\mu)$ the $argmax$ w.r.t the $p$'s? Also, can you explain why $L_{prof}(\bar{x})=n^n$? Should it perhaps be $\prod_i n^{-1} = n^{-n}$? $\endgroup$ – Sameer Jun 25 '12 at 19:14
  • $\begingroup$ @Sameer, the typo is corrected now. However, it is not the argmax. It is the profile likelihood obtained by maximizing the likelihood over all parameter vectors with a given value of $\mu$. Btw with a suitable university access I obtained an electronic version from CRC of the individual chapters in Owen's book. $\endgroup$ – NRH Jun 25 '12 at 19:21

In econometrics, many applied papers start with the assumption that $$ E[g(X,\theta)] = 0 $$ where $X$ is a vector of data, $g$ is a known system of $q$ equations, and $\theta \in \Theta \subseteq \mathbb{R}^p$ is an unknown parameter, $q \geq p$. The function $g$ comes from an economic model. The goal is to estimate $\theta$.

The traditional approach, in econometrics, for estimation and inference on $\theta$ is to use generalized method of moments: $$ \hat{\theta}_\text{GMM} = \text{argmin}_{\theta \in \Theta} \; \bar{g}_n(\theta) 'W \bar{g}_n(\theta) $$ where $W$ is a positive definite weighting matrix and $$ \bar{g}_n(\theta) := \frac{1}{n} \sum_{i=1}^n g(X_i,\theta). $$ Empirical likelihood providers an alternative estimator to GMM. The idea is to enforce the moment condition as a constraint when maximizing the nonparametric likelihood. First, fix a $\theta$. Then solve $$ L(\theta) = \max_{p_1,\ldots,p_n} \; \prod_{i=1}^n p_i $$ subject to $$ \sum_{i=1}^n p_i=1, \qquad p_i \geq 0, \qquad \sum_{i=1}^n p_i \cdot g(X_i,\theta) = 0. $$ This is the `inner loop'. Then maximize over $\theta$: $$ \hat{\theta}_\text{EL} = \text{argmax}_{\theta \in \Theta} \; \log L(\theta). $$ This approach has been shown to have better higher order properties than GMM (see Newey and Smith 2004, Econometrica), which is one reason why it is preferable over GMM. For additional reference, see the notes and lecture by Imbens and Wooldridge here (lecture 15).

There are of course many other reasons why EL has garnered attention in econometrics, but I hope this is a useful starting place. Moment equality models are very common in empirical economics.

  • $\begingroup$ Thank you for writing such a clear, well-referenced answer. Welcome to our community! $\endgroup$ – whuber Jun 27 '12 at 14:12

In survival analysis, the Kaplan-Meier curve is the most famous non-parametric estimator of the survival function $S(t) = Pr(T > t)$, where $T$ denotes the time-to-event random variable. Basically, $\hat{S}$ is a generalisation of the empirical distribution function which allows censoring. It can be derived heuristically, as given in most practical textbooks. But it can also be formally derived as a maximum (empirical) likelihood estimator. Here are more details.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.