# Constrained Maximum Lilkihood Estimation

Given the joint PDF $h_{X,Y}(x,y)$ of the correlated variables $X$ and $Y$, and given that the joint PDF is a function of a parameter $\lambda$ such that $|\lambda|\le 1$, I need to find the maximum likelihood estimator for the parameter $\lambda$. The Joint PDF is given by

$$h_{X,Y}(x,y) = f_X(x)f_Y(y) \left[1+\lambda\left(2F_X(x)-1\right)\left(2F_Y(y)-1\right)\right],\quad |\lambda|\le 1$$

The marginal PDFs $f_X(x)$ and $f_Y(y)$ are known as well as the marginal CDFs $F_X(x)$ and $F_Y(y)$. So how can I apply constrained maximum likelihood estimation to obtain the estimated value of $\lambda$?

• So if you are given i.i.d. observations $(X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)$, can't you just form a likilihood function $L^n(\lambda)$ and then solve the optimization problem: $\max_{\lambda \big| |\lambda| \leq 1} L^n(\lambda)$? Of course, actually maximizing the above function may turn out to be computationally difficult if $L^n(\lambda)$ is not a well-behaved concave function. But aside from that, what is the issue? – Dapz Dec 12 '12 at 11:48
• Thanks Dapz, the issue is that I am not familiar with this type of optimization. Can you please show me how the equations will look like? Also, it will be great if you can point out some references or books with good examples. In fact, I need the liklihood given the observation (x,y). – Remy Dec 18 '12 at 19:44

For a sample of size $n$, $\{x_i,y_i|i=1,...,n\}$ we write for compactness

$$\left(2F_X(x_i)-1\right)\left(2F_Y(y_i)-1\right) \equiv g(x_i,y_i)$$

and we note that $g(x_i,y_i) =0$ if and only if both $x_i$ and $y_i$ equal the medians of the respective marginal distributions. Moreover $0<g(x_i,y_i) <1$ if both are above or below the medians, while $-1<g(x_i,y_i) <0$ if the one is above and the other below the respective median.

The the log-likelihood is

$$\ln L(\lambda \mid \mathbf x, \mathbf y) = \sum_{i=1}^n\ln\left[f_X(x_i)f_Y(y_i)\right] + \sum_{i=1}^n\ln[1+\lambda g(x_i,y_i)]$$

To proceed with the maximization we need to take into account the constraint on the value of $\lambda$. We have

$$|\lambda|\leq 1 \Rightarrow -1\leq \lambda , \lambda \leq 1$$ In so called "normal form" these constraints are written

$$\lambda +1 \geq 0, \;\;1-\lambda \geq 0$$

So the lagrangean of the maximization problem is

$$\Lambda = \ln L(\lambda \mid \mathbf x, \mathbf y) + \xi_1(\lambda +1)+\xi_2(1-\lambda)$$

and we have a standard Karush-Kuhn-Tucker problem of maximization under (linear) inequality constraints), where $\xi_1, \xi_2$ are non-negative multipliers. Note that

$$\frac {\partial \ln L(\lambda \mid \mathbf x, \mathbf y)}{\partial \lambda} = \sum_{i=1}^n\frac{g(x_i,y_i)}{1+\lambda g(x_i,y_i)}$$

and

$$\frac {\partial^2\ln L(\lambda \mid \mathbf x, \mathbf y)}{\partial \lambda^2}=\sum_{i=1}^n\frac{-[g(x_i,y_i)]^2}{(1+\lambda g(x_i,y_i))^2} <0$$

So the log-likelihood is a strictly concave function of $\lambda$. Given also that the constraints are linear, this means that the first-order necessary conditions for a maximum will also be sufficient for a global maximum. The basic condition is

$$\frac {\partial\Lambda}{\partial \lambda} \leq 0\Rightarrow \sum_{i=1}^n\frac{g(x_i,y_i)}{1+\lambda g(x_i,y_i)} +\xi_1-\xi_2 \leq 0$$

The multipliers will be both zero if $\lambda$ is strictly inside $(-1,1)$, and they cannot be both non-zero. Also, given the constraints on the value of $\lambda$ and the range of $g()$, one can find that the sign of the ratio $\frac{g(x_i,y_i)}{1+\lambda g(x_i,y_i)}$ is governed by the sign of $g(x_i,y_i)$, irrespective of the sign of $\lambda$ at the optimum.

This permits as to single out a special case that also has some intuition: For, assume that it so happens that our sample is such that all $g(x_i,y_i)$ are positive, meaning that all pairs of realizations are either either below or above the medians of the marginal distributions. In such a case $\sum_{i=1}^n\frac{g(x_i,y_i)}{1+\lambda g(x_i,y_i)} >0$. Then, given also the non-negativity of the multipliers, the only way that the first-order condition can be satisfied, is for $\xi_2>0$, meaning that the optimal $\lambda$ will be equal to $1$. So a solution here is

$$g(x_i,y_i) >0\; \forall i \Rightarrow \{\lambda^*=1, \xi^*_1=0, \xi^*_2>0\}$$

For the specific joint density function, the sign and value of $\lambda$ is related to the direction and strength respectively, of the dependence between two random variables: if all realizations show that both variables are realized together either above, or below, their medians, this indicates a strong and positive relationship, and this is what the solution tells us.

In general, as is usually the case with problems under inequality constraints, rarely can we derive a final solution analytically, since we have to algorithmically compute the values of the objective function at the various candidate solutions, given the actual sample.