Assume we have a dataset $D=\{(x_i, y_i)\}$ drawn from the joint distribution $(x_i, y_i) \sim P (x=x_i, y=y_i)$. We want to make predictions based on the dataset and for that we use a parametric model with parameter $\theta$ so that the prediction for a new $x^*$ is $P(y^*|x^*, \theta)$.
To estimate $\theta$, we maximize the likelihood $P(D|\theta)$ with respect to $\theta$, i.e. we find $\theta$ that maximizes the likelihood of observing the dataset $D$. To keep things simple, let us assume there is a single data point in the set, $D=\{(x_1, y_1 \}$. Then the likelihood is $P(D|\theta) = P((x_1, y_1)|\theta)$, i.e. the joint probability distribution of $(x_1, y_1)$ conditioned on $\theta$. We can expand the joint distribution as $P((x_1, y_1)|\theta)=P(y_1|x_1,\theta)P(x_1|\theta)$. However, in the usual definition of likelihood, only the first term appears. My question is, what happens to the second term $P(x_1|\theta)$? Is that somehow independent of $\theta$ and thus irrelevant to the estimator? If so, how can I see that?
