Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Suppose I have data $(y_i,x_i)_{i=1}^n$ and I know two density functions $f(y_i;\theta)$ and $g(x_i;\theta)$ -- two density function shares the same parameter $\theta$.
I want to use both information to estimate $\theta$. I consider log likelihood function $$L(\theta;data) = \sum_{i}\log(f(y_i;\theta)) + \log(g(x_i;\theta)).$$
if I maximize this with respect to $\theta$, would this give me a consistent estimator for $\theta$?
Sure, you can do this. It depends on how you think $y$ and $x$ are related. I'll give you an example in which this makes mathematical sense, but is contrived and unrealistic.
If $x$ and $y$ are independent but share a parameter. For instance, suppose $x_i \sim N(1, \theta)$ and $y_i \sim N(-1, \theta)$. Let $x = (x_1,x_2, \ldots, x_n)$ be an iid sample of size from $N(1, \theta)$ and $y = (y_1,y_2, \ldots, y_n)$ be an iid sample of size from $N(-1, \theta)$. Let $\mathcal{L}_x(\theta \mid x)$ be the likelihood function for $x$ and $\ell(\theta \mid x)$ be the log-likelihood function for $x$. We adopt similar notation for $y$.
The likelihood is then
\begin{align} \mathcal{L}(\theta \mid x, y)&= \left\{ \prod_{i = 1}^n \mathcal{L}_x(\theta \mid x_i) \right\} \times \left\{ \prod_{i=1}^n \mathcal{L}_y(\theta \mid y_i)\right\}\\ &=\prod_{i=1}^n \mathcal{L}_x(\theta \mid x_i) \mathcal{L}_y(\theta \mid y_i). \end{align} Using standard properties of logarithms gives us
\begin{align} \ell(\theta \mid x, y) &= \log \mathcal{L}(\theta \mid x, y)\\ &= \log \left\{ \prod_{i=1}^n \mathcal{L}_x(\theta \mid x_i) \mathcal{L}_y(\theta \mid y_i) \right\}\\ &= \sum_{i=1}^n \log \left\{\mathcal{L}_x(\theta \mid x_i) \mathcal{L}_y(\theta \mid y_i) \right\} \\ &= \sum_{i=1}^n \left\{ \log \left[\mathcal{L}_x(\theta \mid x_i)\right] + \left[ \mathcal{L}_y(\theta \mid y_i) \right] \right\}\\ &=\sum_{i=1}^n \left\{ \ell(\theta \mid x_i) + \ell(\theta \mid y_i) \right\} \end{align} In this case I imagine the estimator would be consistent under the usual regularity conditions. That's because we can break the sum up into two sums.
