Assume we have a dataset $D=\{(x_i, y_i)\}$ drawn from the joint distribution $(x_i, y_i) \sim P (x=x_i, y=y_i)$. We want to make predictions based on the dataset and for that we use a parametric model with parameter $\theta$ so that the prediction for a new $x^*$ is $P(y^*|x^*, \theta)$.

To estimate $\theta$, we maximize the likelihood $P(D|\theta)$ with respect to $\theta$, i.e. we find $\theta$ that maximizes the likelihood of observing the dataset $D$. To keep things simple, let us assume there is a single data point in the set, $D=\{(x_1, y_1 \}$. Then the likelihood is $P(D|\theta) = P((x_1, y_1)|\theta)$, i.e. the joint probability distribution of $(x_1, y_1)$ conditioned on $\theta$. We can expand the joint distribution as $P((x_1, y_1)|\theta)=P(y_1|x_1,\theta)P(x_1|\theta)$. However, in the usual definition of likelihood, only the first term appears. My question is, what happens to the second term $P(x_1|\theta)$? Is that somehow independent of $\theta$ and thus irrelevant to the estimator? If so, how can I see that?

  • 2
    $\begingroup$ You wish to predict $y$ based on an observed $x$, so the only thing that matters is $p(y|x, \theta)$. $p(x|\cdot)$ doesn't matter because you will have observed $x$ at the time you are making your prediction, so it is fixed. Your likelihood function needs to reflect that. $\endgroup$
    – jbowman
    Jan 3 at 2:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy