I'm reading (Xu and Knight 2010), which is a paper about estimating finite Gaussian mixture models using the CECF (Continuous Empirical Characteristic Function) method. The basic idea is to minimize a weighted distance between the empirical characteristic function (ECF) and the theoretical one (CF) with respect to parameters of the mixture distribution.

The weighted distance is given by this integral:

$$ D(\theta; \pmb{r}) = \int_{-\infty}^{\infty} |\mathrm{ECF}(t; \pmb{r}) - \mathrm{CF}(t, \theta)|^2 e^{-bt^2} dt $$

Here, $\pmb{r}$ is the vector of data used to compute the empirical characteristic function.

The point of the paper is that this integral can be solved analytically, and the distance $D(\theta; \pmb{r})$ becomes a function that one can compute without integrals. I'll have to put a screenshot here since the function is pretty long:

enter image description here

This function involves the weighting parameter $b$ that influences the asymptotic covariance of the estimate. So I think the distance should really be written $D(\theta; \pmb{r}, b)$.

Asymptotic covariance

By Proposition 2, the estimated mixture parameters $\hat\theta = \arg\min_{\theta} D(\theta; \pmb{r}, b)$ for a fixed $b$ are asymptotically normal with the following covariance matrix:

$$ \begin{cases} \mathrm{Var}(\hat\theta) &= \Lambda^{-1}\Omega\Lambda^{-1}\\ \Lambda_{ij} &= \mathbb{E} \dfrac{ \partial D^2(\theta; \pmb{r}, b) }{ \partial \theta_i \partial \theta_j }\\ \Omega_{ij} &= \mathbb{E} \left( \dfrac{\partial D(\theta; \pmb{r}, b)}{\partial \theta_i} \dfrac{\partial D(\theta; \pmb{r}, b)}{\partial \theta_j} \right) \end{cases} $$


How does one compute these expectations?

To me this looks similar to the Fisher information matrix which one estimates using the Hessian of the log-likelihood at the estimate $\hat\theta$. This makes sense because the log-likelihood is a sum, so when one differentiates it, they get a sum of derivatives (or Hessians), and this sum is basically the average ("empirical expectation").

But here $D(\theta; \pmb{r}, b)$ isn't just a sum over the data $\pmb r$. In fact, the first quantity is this double nested sum over the data, and the last quantity involves $K$ sums over the data. So, the result doesn't look like the average of anything to me.

Am I overthinking this, and is $\Lambda$ just the Hessian of $D$ evaluated at the estimates $\hat\theta$?

How do I compute $\Omega_{ij}$? I was thinking about computing the derivatives $\partial D(\theta; r_n, b) / \partial\theta_i$ for each data point $r_n \in \pmb r$, then calculating this sum:

$$ \Omega_{ij} \approx \frac1N \sum_{n=1}^N \left[ \frac{ \partial D(\theta;r_n,b) }{\partial\theta_i} \frac{ \partial D(\theta;r_n,b) }{\partial\theta_j} \right] $$

Is this the right approach?

Proof of Lemma 1 in the Appendix says:

Recall that $\Omega_{ij} = \int \left(\frac{\partial D(\theta;r)}{\partial\theta_i} \frac{\partial D(\theta;r)}{\partial\theta_j}\right) \exp(-bt^2) dt$

...which seems weird because $D(\theta;r)$ doesn't depend on $t$, so one can move the product of derivatives out of the integral and compute $\left(\frac{\partial D(\theta;r)}{\partial\theta_i} \frac{\partial D(\theta;r)}{\partial\theta_j}\right) \int \exp(-bt^2) dt$, but this isn't an expectation, is it?




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.