0
$\begingroup$

Let $\{X_i\}$ a sequence of independent random variables. Each $X_i$ has a p.d.f $p(m, \theta)$. Where $\theta$ is a real unknown parameter and $m$ the outcome of another random variable $M$ with p.d.f $p(m)$.

I have the next protocol to get a sample $\bar{x}=\{x_1,...,x_n\}$:

First, I get the value $m_1 \sim p(m)$. Then I obtain the value $x_1 \sim p(m_1 \theta)$. And I repeat the process $n$-times.

The aim is to estimate the value of $\theta$ from $\bar{x}$.

My questions:

  1. the maximum likelihood estimator is the best way to do that?
  2. Are there lower bound inequalities for the variance of the estimator for $\theta$?

When $m$ and $\theta$ are real unknown parameters, I know that there is the Cramér-Rao Bound, and when one has a parameter that is a random variable there exists the Van Trees inequality. But in this situation, I don't know if there is a standard inequality.

$\endgroup$
1
$\begingroup$

Regarding your questions:

  1. I think there is a confusion: it does not really make sense to ask if the MLE is a "good way" to estimate $\theta$. As its name suggests, the MLE is your best estimate of $\theta$ that maximizes the likelihood of your observation, so yes, it is pretty good ^^ But a more relevant question is how to obtain your MLE. Given that you have both observed variables $X_i$ and hidden variables $M_i$, I would suggest using the Expectation-Maximization algorithm.
  2. You can indeed use the Cramér-Rao bound. In this case, the likelihood used to compute the Fisher Information is the likelihood of your observations $X_i$, obtained by marginalizing the joint distribution of the observed and hidden variables:

$$ p(X_i|\theta) = \sum_{M_i}p(X_i|M_i,\theta)p(M_i) $$

I highly recommend this article, in which the authors use the EM algo to obtain the MLE of a vector of parameters $\theta$, and also compute the Fisher Information matrix: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4867027/

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The (complete information) likelihood in this model is$$\prod_i p(x_i|m_i,\theta)\tag{1}$$which can be maximised or used in a Bayesian analysis, providing a prior $p(\theta)$ is chosen. (There is no "best way to estimate" a quantity, it all depends on the utility function for running the estimation.)

In the event the $m_i$'s are not observed, the (observed) likelihood becomes$$\prod_i \int p(x_i|m,\theta)\,p(m)\,\text{d}m\tag{2}$$

If there exists an unbiased estimator of $\theta$ in this setting, $T_n(X_1,\ldots,X_n|m_1,\ldots,m_n)$ then it satisfies the Cramér-Rao inequality$$\text{cov}(T_n(X_1,\ldots,X_n|m_1,\ldots,m_n)) \ge I_n^{-1}(\theta;m_1,\ldots,m_n)$$where $I_n(\theta;m_1,\ldots,m_n)$ is the (complete) Fisher information matrix associated with (1): $$I_n(\theta;m_1,\ldots,m_n)=-\sum_{i=1}^n\mathbb{E}_\theta \left[ \dfrac{\partial ^2}{\partial \theta \, \partial \theta^\text{T}} \log p\left(X| m_i,{\theta}\right)\right]$$ Once again, if the $m_i$'s are not observed, the observed Fisher information become $$I_n(\theta)=-n \mathbb{E}_\theta \left[ \dfrac{\partial ^2}{\partial \theta \, \partial \theta^\text{T}} \log p\left(X| {\theta}\right)\right]=-n \mathbb{E}_\theta \left[ \dfrac{\partial ^2}{\partial \theta \, \partial \theta^\text{T}} \log \int p\left(X| m,{\theta}\right)p(m)\,\text{d}m \right]$$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I am not absolutely sure about the way you define the likelihood in the model. Here, you seem to consider the conditional likelihood of $x_i$ given $m_i$. But I guess the original question implies that $m_i$ is a hidden variable, to which we do not have access. If this is the case, we should define the likelihood of the observations as $p(x_i|\theta) = \sum_{m_i}p(x_i|m_i,\theta)p(m_i)$ $\endgroup$ – Camille Gontier May 9 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.