19
$\begingroup$

Estimating parameters using maximum likelihood estimation (MLE) involves evaluating the likelihood function, which maps the probability of the sample (X) occurring to values (x) on the parameter space (θ) given a distribution family (P(X=x|θ) over possible values of θ (note: am I right on this?). All examples I've seen involve calculating P(X=x|θ) by taking the product of F(X) where F is the distribution with the local value for θ and X is the sample (a vector).

Since we're just multiplying the data, does it follow that the data be independent? E.g. could we not use MLE to fit time-series data? Or do the parameters just have to be independent?

$\endgroup$

3 Answers 3

16
$\begingroup$

The likelihood function is defined as the probability of an event $E$ (data set ${\bf x}$) as a function of the model parameters $\theta$

$${\mathcal L}(\theta;{\bf x})\propto {\mathbb P}(\text{Event }E;\theta)= {\mathbb P}(\text{observing } {\bf x};\theta).$$

Therefore, there is no assumption of independence of the observations. In the classical approach there is no definition for independence of parameters since they are not random variables; some related concepts could be identifiability, parameter orthogonality, and independence of the Maximum Likelihood Estimators (which are random variables).

Some examples,

(1). Discrete case. ${\bf x}=(x_1,...,x_n)$ is a sample of (independent) discrete observations with ${\mathbb P}(\text{observing } x_j ; \theta)>0$, then

$${\mathcal L}(\theta;{\bf x})\propto \prod_{j=1}^n{\mathbb P}(\text{observing } x_j ; \theta).$$

Particularly, if $x_j\sim \text{Binomial}(N,\theta)$, with $N$ known, we have that

$${\mathcal L}(\theta;{\bf x})\propto \prod_{j=1}^n \theta^{x_j}(1-\theta)^{N-x_j}.$$

(2). Continuous approximation. Let ${\bf x}=(x_1,...,x_n)$ be a sample from a continuous random variable $X$, with distribution $F$ and density $f$, with measurement error $\epsilon$, this is, you observe the sets $(x_j-\epsilon,x_j+\epsilon)$. Then

\begin{eqnarray*} {\mathcal L}(\theta;{\bf x})\propto \prod_{j=1}^n {\mathbb P}[\text{observing } (x_j-\epsilon,x_j+\epsilon);\theta] = \prod_{j=1}^n[F(x_j+\epsilon;\theta)-F(x_j-\epsilon;\theta)] \end{eqnarray*}

When $\epsilon$ is small, this can be approximated (using the Mean Value Theorem) by

\begin{eqnarray*} {\mathcal L}(\theta;{\bf x})\propto \prod_{j=1}^n f(x_j;\theta) \end{eqnarray*}

For an example with the normal case, take a look at this.

(3). Dependent and Markov model. Suppose that ${\bf x}=(x_1,...,x_n)$ is a set of observations possibly dependent and let $f$ be the joint density of ${\bf x}$, then

\begin{eqnarray*} {\mathcal L}(\theta;{\bf x})\propto f({\bf x}; \theta). \end{eqnarray*}

If additionally the Markov property is satisfied, then

\begin{eqnarray*} {\mathcal L}(\theta;{\bf x})\propto f({\bf x}; \theta) = f(x_1;\theta)\prod_{j=1}^{n-1} f(x_{j+1} \vert x_j ;\theta). \end{eqnarray*}

Take also a look at this.

$\endgroup$
2
  • 5
    $\begingroup$ From the you write the likelihood function as a product, you are implicitly assuming a dependence structure among the observations. So for MLE one needs two assumptions (a) one on the distribution of each individual outcome and (b) one on the dependence among the outcomes. $\endgroup$
    – user83346
    Nov 9, 2017 at 7:01
  • 1
    $\begingroup$ This is a great answer. However I think it would be more clear to readers if you stated that although the likelihood function itself is not concerned with sample independence, the probability model you use certainly is (as per example 3). Many readers may gloss over the details you have included. $\endgroup$
    – Chris
    Sep 20, 2021 at 1:09
13
$\begingroup$

(+1) Very good question.

Minor thing, MLE stands for maximum likelihood estimate (not multiple), which means that you just maximize the likelihood. This does not specify that the likelihood has to be produced by IID sampling.

If the dependence of the sampling can be written in the statistical model, you just write the likelihood accordingly and maximize it as usual.

The one case worth mentioning when you do not assume dependence is that of the multivariate Gaussian sampling (in time series analysis for example). The dependence between two Gaussian variables can be modelled by their covariance term, which you incoroporate in the likelihood.

To give a simplistic example, assume that you draw a sample of size $2$ from correlated Gaussian variables with same mean and variance. You would write the likelihood as

$$\frac{1}{2\pi\sigma^2\sqrt{1-\rho^2}}\exp\left(-\frac{z}{2\sigma^2(1-\rho^2)}\right),$$

where $z$ is

$$z = (x_1-\mu)^2-2\rho(x_1-\mu)(x_2-\mu)+(x_2-\mu)^2.$$

This is not the product of the individual likelihoods. Still, you would maximize this with parameters $(\mu, \sigma, \rho)$ to get their MLE.

$\endgroup$
4
  • 3
    $\begingroup$ These are good answers and examples. The only thing I would add to see this in simple terms is that likelihood estimation only requires that a model for the generation of the data be specified in terms of some unknown parameters be described in functional form. $\endgroup$ May 26, 2012 at 11:53
  • $\begingroup$ (+1) Absolutely true! Do you have an example of model that cannot be specified in those terms? $\endgroup$
    – gui11aume
    May 26, 2012 at 12:17
  • $\begingroup$ @gu11aume I think you are referring to my remark. I would say that I was not giving a direct answer to the question. The answwer to the question is yes because there are examples that can be shown where the likelihood function can be expressed when the data are genersted by dependent random variables. $\endgroup$ May 26, 2012 at 13:07
  • 2
    $\begingroup$ Examples where this cannot be done would be where the data are given without any description of the data generating mechanism or the model is not presented in a parametric form such as when you are given two iid data sets and are asked to test whether they come from the same distribution where you only specify that the distributions are absolutely continuous. $\endgroup$ May 26, 2012 at 13:08
4
$\begingroup$

Of course, Gaussian ARMA models possess a likelihood, as their covariance function can be derived explicitly. This is basically an extension of gui11ame's answer to more than 2 observations. Minimal googling produces papers like this one where the likelihood is given in the general form.

Another, to an extent, more intriguing, class of examples is given by multilevel random effect models. If you have data of the form $$y_{ij} = x_{ij}'\beta + u_i + \epsilon_{ij},$$ where indices $j$ are nested in $i$ (think of students $j$ in classrooms $i$, say, for a classic application of multilevel models), then, assuming $\epsilon_{ij} \perp u_i$, the likelihood is $$ \ln L \sim \sum_i \ln \int \prod_j f(y_{ij}|\beta,u_i) {\rm d}F(u_i) $$ and is a sum over the likelihood contributions defined at the level of clusters, not individual observations. (Of course, in the Gaussian case, you can push the integrals around to produce an analytic ANOVA-like solution. However, if you have say a logit model for your response $y_{ij}$, then there is no way out of numerical integration.)

$\endgroup$
1
  • 2
    $\begingroup$ Stask and @gui11aume, these three answers are nice but I think they miss a point: what about the consistency of the MLE for dependent data ? $\endgroup$ Feb 22, 2013 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.