1
$\begingroup$

The inference based on minimizing the power divergence $$D_{\lambda}(g\|f) = \frac{1}{\lambda - 1} \log \int g^{\lambda} f^{1-\lambda} dx$$ is known to be robust against outliers for $\lambda <1$. Here $g$ is the data driven density and $f\in \mathcal{F}=\{f_\theta\}_{\theta\in \Theta}$ is the model density. $\lambda =1$ corresponds to the Maximum Liklihood Estimation (MLE), since $D_\lambda$ converges to the KL-divergence as $\lambda\to 1$.

So the inference based on $D_\lambda$ can be regarded as a generalization of MLE.

I am wondering whether there is an estimating equation corresponding to this inference as a generalization of the estimating equation of MLE. The estimating equation for MLE is $$\sum_{i=1}^n s(X_i;\theta) = 0,$$ where $s(\cdot,\theta) = \nabla_\theta\log f_\theta(\cdot)$ is the score function; $X_1,\dots,X_n$ are the data points assumed to have drawn from $f_\theta$; $\nabla_\theta$ stands for gradient w.r.to $\theta$.

$\endgroup$
2
  • $\begingroup$ I admit I don't know anything about power divergence, but I don't see how lambda=1 corresponds to MLE, since setting lambda of 1 gives us a denominator of 0 in the leading coefficient! Are you sure you've correctly specified D? A quick Google search for "power divergence" brings up some similar but markedly different forms for the integral, though I don't know enough about this type of analysis to know whether those are applicable. $\endgroup$ – Ryan Simmons Sep 3 '15 at 13:32
  • $\begingroup$ @RyanSimmons: I meant actually $\lambda \to 1$. Yes, there are several forms for the power divergence. But they all can be shown to be equivalent upto a monotone function. As we are concerned about minimization, it doesn't really matter. $\endgroup$ – Ashok Sep 3 '15 at 13:46
1
+50
$\begingroup$

Let $X_i$ be an i.i.d sample from a distribution with p.d.f $x \rightarrow f(x;\theta)$.

Maximum likelihood amounts to minimizing the KL divergence of the parametric distribution $p_{\theta}(x)$ to the empirical data distribution $p_D(x) = \frac{1}{n} \sum_{i=1}^n \delta_{X_i}(x)$.

Adapting this principle to the power divergence, we want to minimize $$\frac{1}{\lambda-1}\log \int_{-\infty}^{\infty} \left(\frac{1}{n}\sum_{i=1}^n \delta_{X_i}(x)\right)^{\lambda} p_{\theta}(x)^{1-\lambda} ~\textrm{d}x$$

This doesn't make a whole lot of sense since we're taking the power of a distribution which isn't defined by a pdf, but very loosely we can intuit this is equivalent to minimizing

$$\sum_{i=1}^n p_{\theta}(X_i)^{1-\lambda}$$

thus setting

$$\sum_{i=1}^n \left(\frac{\partial}{\partial\theta}p(X_i;\theta)\right)p_{\theta}(X_i)^{-\lambda} = 0$$

$\endgroup$
8
  • $\begingroup$ Thanks for the response. Is there a reference for what you said? Especially I am not very convinced about "....This doesn't make a whole lot of sense since we're taking the power of a distribution which isn't defined by a pdf, but very loosely we can intuit this is equivalent to minimizing..." Is there any justification that one can do this? $\endgroup$ – Ashok Sep 3 '15 at 12:44
  • $\begingroup$ I think you would have to go through the Radon–Nikodym derivative. But at best, this will tell you whether or not the equation is legitimate. If it is, it can't really be anything else than what I've derived... the support has to be the $X_i$, and the weight on the $X_i$ has to be symmetric... $\endgroup$ – Arthur B. Sep 3 '15 at 14:04
  • $\begingroup$ Can you elaborate a bit? $\endgroup$ – Ashok Sep 3 '15 at 14:29
  • $\begingroup$ There is no other reasonable way to define it. The technique can't invent new data points ex nihilo or privilege some over the others $\endgroup$ – Arthur B. Sep 3 '15 at 14:30
  • $\begingroup$ //The technique can't invent new data points ex nihilo or privilege some over the others// I couln't get this. $\endgroup$ – Ashok Sep 3 '15 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.