Derivation of Joint Likelihood with Only Conditional Independence Assumptions (Without Marginal Independence)

Question

I'm working through a derivation of the joint likelihood for a dataset $(\mathbf{X}_i, \mathbf{y}_i)$ under the assumption of conditional independence of $\mathbf{y}_i$ given $\mathbf{X}_i$, without assuming marginal independence of the features $\mathbf{X}_i$. I’m trying to confirm whether my understanding of the joint likelihood expression is correct.

Here’s the setup:

1. I have $N$ observations where $\mathbf{y}_i$ is the target variable and $\mathbf{X}_i$ is the feature vector of observation $i$.
2. I assume conditional independence of $\mathbf{y}_i$ given $\mathbf{X}_i$. This means:

$$ p(y_1, y_2, \ldots, y_N \mid \mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N) = \prod_{i=1}^N p(y_i \mid \mathbf{X}_i). $$

3. I do not assume any marginal independence among $\mathbf{X}_i$ values, so the joint distribution $p(\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N)$ could be arbitrary.

Given this, I derive the joint likelihood of observing both the target values and the features as follows:

Derivation Attempt:

Using the chain rule, I write the joint probability as:

$$ p(y_1, y_2, \ldots, y_N, \mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N;\theta) = p(y_1, y_2, \ldots, y_N \mid \mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N;\theta) \cdot p(\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N). $$

Applying conditional independence of $\mathbf{y}_i$'s given $\mathbf{X}_i$'s, I replace the conditional joint probability with:

$$ p(y_1, y_2, \ldots, y_N \mid \mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N;\theta) = \prod_{i=1}^N p(y_i \mid \mathbf{X}_i;\theta). $$

Thus, the joint likelihood becomes:

$$ L(\theta) = p(y_1, y_2, \ldots, y_N, \mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N) = \left( \prod_{i=1}^N p(y_i \mid \mathbf{X}_i; \theta) \right) \cdot p(\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N). $$

Questions:

1. Is this derivation correct for the joint likelihood under the assumptions given?
2. When maximizing the likelihood for $\theta$, am I correct to focus only on the conditional part $\prod_{i=1}^N p(y_i \mid \mathbf{X}_i; \theta)$ since $p(\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_N)$ does not depend on $\theta$?

Any feedback on this derivation or references to similar derivations in statistical literature would be greatly appreciated!

Answer 1

Your derivation represents a frequentist MLE approach to estimate parameters from scratch via the full joint distribution. Here since you assume the critical conditional independence of all $\mathbf{y}_i$ given all their respective $\mathbf{X}_i$, your above derivation is sound so long as an additional critical implicit assumption that each data pair follows a shared discriminative model $p(\mathbf{y}_i|\mathbf{X}_i)$ is added.

Your formulation is essentially equivalent to the textbook linear regression's i.i.d. assumption for irreducible errors of all data pair observations with the additional fact that in linear regression observations of all $\mathbf{X}_i$ are fixed constants (no longer random variables to be conditioned on) and your conditional distribution $p(\mathbf{y}_i|\mathbf{X}_i;\theta)$ becomes marginal distribution over the parameters alone.