1
$\begingroup$

Suppose that we have an $i.i.d$ sample, $\{Y_i,X_i\}_{i=1}^N$, and a correctly specified conditional density of $Y$ given $X$, $f(Y|X; \theta)$, where $\theta$ is the parameters of the density.

Then, the MLE theory show that the true value of the parameters, $\theta_0$, maximize the expected value, $\mathbb{E}\left[f(Y|X;\theta)\right]$.

That is, $\theta_0=\arg\max_\limits{\theta\in\Theta} \mathbb{E}\left[f(Y|X;\theta)\right]$, where $\Theta$ is the compact parameter space.

Therefore, we use the maximum likelihood estimator, $\widehat{\theta}_{MLE}=\arg\max_\limits{\theta\in\Theta}N^{-1}\sum_{i=1}^Nf(Y_i|X_i;\theta)$.

Here, I am wondering what will happens if we use only a subsample that has a specific value of a random variable: $\{Y_i,X_i:D_i=1\}$, where $D$ is a random variable indicating whether $i$ is included in the subsample or not.

In particular, the resulting M-estimator will be $\widehat{\theta}_{sub}=\arg\max_\limits{\theta\in\Theta}\sum_\limits{i:D_i=1}f(Y_i|X_i;\theta)$.

In my opinion, $\widehat{\theta}_{sub}$ may converge to a parameter value $\theta_*=\arg\max_\limits{\theta\in\Theta}\mathbb{E}\left[f(Y|X;\theta)|D=1\right]$.

Since $\mathbb{E}\left[f(Y|X;\theta)\right]=\Pr[D=1]\mathbb{E}[f(Y|X;\theta)| D=1]+\Pr[D=0]\mathbb{E}[f(Y|X;\theta)| D=0]$, there is no reason $\theta_*$ is in fact the true value $\theta_0$.

Then, what is the condition that is required for $\theta_0=\theta_*$?

$\endgroup$

1 Answer 1

1
$\begingroup$

It is clearly sufficient that $Y$ is independent of $D$ given $X$, since then $E[f(Y|X;\theta)|D]=E[f(Y|X;\theta)]$ and $$\arg\max_\theta E[f(Y|X;\theta)|D]=\arg\max_\theta E[f(Y|X;\theta)]$$

A weaker condition will depend on the model. For example, if the model were a classical linear regression model with known variance, $Y\sim N(X\beta,1)$, it would work to choose $D$ so that the distributions $Y|X=x$ were all symmetrically trimmed of outliers. But while that would give $\beta_0=\beta^*$ in this model, it would no longer work after expanding the model to $Y\sim N(X\beta,\sigma^2)$ since $\sigma^2$ would be underestimated.

You can see from this example that preserving $\beta^*$ requires the distribution of $D$ to depend on $\beta$, not just on $Y$ and $X$ -- the observations you are removing are those with large values of $|Y-X\beta|$. If you assume that the relationship between $(X,Y)$ and $D$ does not depend on $\theta$ and you require $\theta_0=\theta^*$ for all values of $\theta_0\in \Theta$ then I think it's necessary to have $D$ (asymptotically) independent of $Y$ given $X$

$\endgroup$
1
  • $\begingroup$ Thank you for your contribution. $\endgroup$ Commented Jun 27, 2023 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.