What is the distribution of the (arbitrarily) weighted Maximum Likelihood Estimator?

Question

Suppose you observe vector $X_i$ of independent variables, and $y_i$ dependent variables, with likelihood $l\left(\theta;X_i,y_i\right)$. Assume the $y_i$ are independent. Furthermore assume you are given positive weights, $w_i$ which are arbitrary, and compute the weighted Maximum Likelihood Estimator (WMLE?): $$ \hat{\theta} = \arg \max_{\theta} \sum_{1\le i\le n} w_i \log l\left(\theta;X_i,y_i\right). $$ What is the distribution of the WMLE, $\hat{\theta}$?

If I may complicate the question further without splitting it in two, there are two cases to consider:

1. The $w_i$ are completely independent of the $X_i$ and $y_i$.
2. The $w_i$ depend on the dependent variable $y_i$ in some way (perhaps deterministic or stochastic.)

Answer 1

In general, your question has no answer. There are a couple of reasons.

1) Suppose that all $w_i = 1$. Even in that case, the distribution of MLE estimate depends on the distribution of data, i.e. on the function $l(\theta;x,y)$. For instance, it is possible to prove that in the exponential family of distributions, combined with a few more restrictions, the MLE estimate is asymptotically Normal. However, once $l(\theta;x,y)$ is outside the exponential family, anything can happen.

2) Even if $l(\theta;x,y)$ is in the exponential family, the presence of weights (especially if they are dependent on x and Y) is very likely to make the asymptotic distribution results invalid.

Answer 2

In general the answer of Nik Tuzov is correct, but some details are not completely correct. In summary, the distribution of the WMLE is unknown. You can write down the actual equation for the MLE (weights or none) and write the complete derivative to determine the extremal point(s) maximum. Which gives you a computational answer -- but without the specific knowledge of the underlying distribution, you can't perform it.

Actually the presence of the weights doesn't change the question much, since you still just have to compute the derivative. The typical usage of LE in applied science is exactly with weights that depend on the Y -- think counting experiments/results distributed as Poissonian with associated uncertainties that act as weights.

In the practical application, where the LE is performed numerically, a typical approximation is a parabolical shape around the maximum value. You can interpret this either as "normal distribution" or as the first non-vanishing element of the Taylor expansion. But apart from special cases it's not accurate (and can be determined much better even numerically).

So: in simple cases for the underlying distribution you might be able to derive an analytical description for the resulting distribution -- where the series actually converges. Otherwise: no, so in general also: no.