# Parameter estimation for random variables where a control parameter is another r.v

Let $$\{X_i\}$$ a sequence of independent random variables. Each $$X_i$$ has a p.d.f $$p(m, \theta)$$. Where $$\theta$$ is a real unknown parameter and $$m$$ the outcome of another random variable $$M$$ with p.d.f $$p(m)$$.

I have the next protocol to get a sample $$\bar{x}=\{x_1,...,x_n\}$$:

First, I get the value $$m_1 \sim p(m)$$. Then I obtain the value $$x_1 \sim p(m_1 \theta)$$. And I repeat the process $$n$$-times.

The aim is to estimate the value of $$\theta$$ from $$\bar{x}$$.

My questions:

1. the maximum likelihood estimator is the best way to do that?
2. Are there lower bound inequalities for the variance of the estimator for $$\theta$$?

When $$m$$ and $$\theta$$ are real unknown parameters, I know that there is the Cramér-Rao Bound, and when one has a parameter that is a random variable there exists the Van Trees inequality. But in this situation, I don't know if there is a standard inequality.

## 2 Answers

Regarding your questions:

1. I think there is a confusion: it does not really make sense to ask if the MLE is a "good way" to estimate $$\theta$$. As its name suggests, the MLE is your best estimate of $$\theta$$ that maximizes the likelihood of your observation, so yes, it is pretty good ^^ But a more relevant question is how to obtain your MLE. Given that you have both observed variables $$X_i$$ and hidden variables $$M_i$$, I would suggest using the Expectation-Maximization algorithm.
2. You can indeed use the Cramér-Rao bound. In this case, the likelihood used to compute the Fisher Information is the likelihood of your observations $$X_i$$, obtained by marginalizing the joint distribution of the observed and hidden variables:

$$p(X_i|\theta) = \sum_{M_i}p(X_i|M_i,\theta)p(M_i)$$

I highly recommend this article, in which the authors use the EM algo to obtain the MLE of a vector of parameters $$\theta$$, and also compute the Fisher Information matrix: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4867027/

The (complete information) likelihood in this model is$$\prod_i p(x_i|m_i,\theta)\tag{1}$$which can be maximised or used in a Bayesian analysis, providing a prior $$p(\theta)$$ is chosen. (There is no "best way to estimate" a quantity, it all depends on the utility function for running the estimation.)

In the event the $$m_i$$'s are not observed, the (observed) likelihood becomes$$\prod_i \int p(x_i|m,\theta)\,p(m)\,\text{d}m\tag{2}$$

If there exists an unbiased estimator of $$\theta$$ in this setting, $$T_n(X_1,\ldots,X_n|m_1,\ldots,m_n)$$ then it satisfies the Cramér-Rao inequality$$\text{cov}(T_n(X_1,\ldots,X_n|m_1,\ldots,m_n)) \ge I_n^{-1}(\theta;m_1,\ldots,m_n)$$where $$I_n(\theta;m_1,\ldots,m_n)$$ is the (complete) Fisher information matrix associated with (1): $$I_n(\theta;m_1,\ldots,m_n)=-\sum_{i=1}^n\mathbb{E}_\theta \left[ \dfrac{\partial ^2}{\partial \theta \, \partial \theta^\text{T}} \log p\left(X| m_i,{\theta}\right)\right]$$ Once again, if the $$m_i$$'s are not observed, the observed Fisher information become $$I_n(\theta)=-n \mathbb{E}_\theta \left[ \dfrac{\partial ^2}{\partial \theta \, \partial \theta^\text{T}} \log p\left(X| {\theta}\right)\right]=-n \mathbb{E}_\theta \left[ \dfrac{\partial ^2}{\partial \theta \, \partial \theta^\text{T}} \log \int p\left(X| m,{\theta}\right)p(m)\,\text{d}m \right]$$

• I am not absolutely sure about the way you define the likelihood in the model. Here, you seem to consider the conditional likelihood of $x_i$ given $m_i$. But I guess the original question implies that $m_i$ is a hidden variable, to which we do not have access. If this is the case, we should define the likelihood of the observations as $p(x_i|\theta) = \sum_{m_i}p(x_i|m_i,\theta)p(m_i)$ – Camille Gontier May 9 at 9:24