# Regression and independent random vectors

Lets consider that data samples are generated from random vectors $$(X_1, Y_1)...(X_N, Y_N)$$ of cross-sectional data. For regression one usually assumes that the error distribution is I.I.D. normally distributed and the likelihood of data is defined purely by the conditional distribution (the marginal distribution of the $$X_i$$'s are ignored):

$$L(D;\theta) = p_{Y|X}(y_1|x_1;\theta)p_{Y|X}(y_2|x_2;\theta)...p_{Y|X}(y_N|x_N;\theta)$$

Where $$p_{Y|X}$$ is the probability function for conditional distribution.

This leads me to make the following conclusions:

1. The conditional p.d.f. $$p_{Y|X}(y_i|x_i)$$ of each random vector is independent from the other ones. This can only be possible if the random vectors $$(X_1, Y_1)...(X_N, Y_N)$$ are also independent as any function of independent random vectors are also independent.

2. $$p_{Y|X}$$ - is a fixed function that models conditional distribution for data samples.

3. Regression does not make any assumption on the joint distribution of random variables within each random vector of the DGP, each random vector could each have distinct joint distributions i.e - $$p_{X_1,Y_1} ,p_{X_2,Y_2}...p_{X_N,Y_N}$$ The only assumption is they are independent of each other.

Do these make sense?

I think perhaps you have some fuzziness about conditional independence and its relationship to independence. I'll try to clarify this by explaining the scope of regression analysis and the standard conditional independence assumption that leads to the core model form.

Following your question, suppose you have some random vectors $$(X_1,Y_1),...,(X_n,Y_n)$$ that you want to model. If you model the joint distribution of both random variables then that falls within the field of multivariate analysis. However, if you are only interested in modelling the conditional distribution of the "response variable" given the "explanatory variable" then that is a regression analysis. For the latter case, under the assumption that $$Y_1,...,Y_n$$ are conditionally independent given the corresponding explanatory variables $$X_1,...,X_n$$ and some parameter $$\theta$$, you get a sampling density of the form:

$$p(\mathbf{y}|\mathbf{x},\theta) = \prod_{i=1}^n p_{Y|X,\theta}(y_i|x_i, \theta),$$

where $$p_{Y|X,\theta}$$ is the conditional density for the response variable. The goal of regression analysis is to use the data to make inferences about the conditional density $$p_{Y|X,\theta}$$ and resulting predictions, etc. You likewise have a corresponding likelihood function of the form:

$$L_{\mathbf{y},\mathbf{x}}(\theta) \overset{\theta}{\propto} p(\mathbf{y}|\mathbf{x},\theta) = \prod_{i=1}^n p_{Y|X,\theta}(y_i|x_i, \theta).$$

You can find some discussion of the relationship between marginal and conditional independence, and the implications of the latter, in O'Neill (2009). I recommend you have a look at this paper to flesh out your understanding of conditional independence and its implications. Now, with regard to your specific observations:

• (1a) It is not accurate to refer to the density $$p_{Y|X,\theta}$$ as independent or non-independent of anything --- it is the underlying random variables that have conditional independence relations, whereas the conditional density is not a random variable and so it does not have these relations.

• (1b) It is not true that conditional independence also requires full independence of the joint random vector. Indeed, it is entirely possible that the random variables $$X_1,...,X_n$$ might be statistically dependent but you still have the conditional independence for $$Y_i|X_i$$ used in regression analysis. That circumstance would give rise to dependence between the joint random vectors $$(X_i,Y_i)$$ in the analysis.

• (2) The density $$p_{Y|X,\theta}$$ is going to be a fixed function even in the absence of the conditional independence assumption. This is a conditional density that will exist under broad circumstances, irrespective of whether the underlying values are conditionally independent. If the underlying variables are not conditionally independent then you won't be able to write the sampling density as a product of individual conditioning densities for each datapoint, but those conditional densities are still well-defined.

• (3) That is correct --- regression analysis is undertaken conditional on the explanatory variables and it does not make any assumptions about the joint distribution of the response and explanatory variables taken together. When we move into that territory we are in the field of multivariate analysis, which goes beyond regression analysis.

• Thank you. I agree there is an issue with my notation for independence and what you have said above makes sense. Given that a sequence of random vectors are INID does this derivation make sense? What I am trying to understand is if we have cross-sectional data as mentioned in the question (hence no dependence between the random vectors in the sequence), can we apply regression if the random vectors are non identically distributed? Seems to me the answer is yes. Commented Aug 1 at 9:47
• The reason I started with the joint distribution in my answer can be found in a couple of books that I have linked stats.stackexchange.com/questions/651795/… Commented Aug 1 at 9:50

For cross-sectional data, MLE is formulated on the joint distribution of independent random vectors - $$(X_1,Y_1),(X_2,Y_2)...(X_N,Y_N)$$ each could be having a distinct pdf:

$$L(D;\theta) = p_{X_1,Y_1}(x_1,y_1;\theta)p_{X_2,Y_2}(x_2,y_2;\theta)...p_{X_N,Y_N}(x_N,y_N;\theta)$$

Factoring out the marginals we get:

$$L(D;\theta) = p_{Y_1|X_1}(y_1|x_1;\theta)p_{X_1}(x_1)p_{Y_2|X_2}(y_2|x_2;\theta)p_{X_2}(x_2)...p_{Y_N|X_N}(y_N|x_N;\theta)p_{X_N}(x_N)$$

Taking log on both sides:

$$\log(L(D;\theta)) = (\log(p_{Y_1|X_1}(y_1|x_1;\theta)) + \log(p_{Y_2|X_2}(y_2|x_2;\theta)))… + (\log(p_{X_1}(x_1)) + \log(p_{X_2}(x_2)))…$$

Ignoring the marginals as they have no effect on the regression model parameters:

$$\log(L(D;\theta)) = (\log(p_{Y_1|X_1}(y_1|x_1;\theta)) + \log(p_{Y_2|X_2}(y_2|x_2;\theta))) …$$

Assuming all the conditionals can be described using the same function as this is mostly expected from NN or ML based regression models:

$$\log(L(D;\theta)) = (\log(p_{Y|X}(y_1|x_1;\theta)) + \log(p_{Y|X}(y_2|x_2;\theta)))…$$

Hence the only requirement is for the random vectors generating the data to be independent (no restriction on identical) and that the conditional distribution can be properly described by a single function in $$(x_i,y_i)$$. This is tested by the IID'ness of the error distribution.

• As I mentioned above, your notation for joint pdf is mixed up and non-standard, you can check my references carefully. As for your another linked unanswered post I'll try to answer later once I have time. But pls note in linear and logistic regressions we implicitly assume all cross-sectional data pairs are IID from a same fixed joint distribution as stated in my above answer. This way the NLL is a simple sum of log functions which is convex and easy to optimize with linear models assumption Commented Jul 28 at 1:07
• The only assumption for linear regression is for the error distribution to be IID. It does not make any distribution on the distribution of the DGP. Can you tell me what is incorrect with the above derivation? It does not assume "different conditional distributions" as commented below. Commented Jul 28 at 17:37
• I've submitted necessary confused notation correction for your answer, and after that you can clearly see your problem: in general the factorization p(x,y;θ)=p(y∣x;θ)p(x) implies that the marginal distribution p(x) must be independent of θ. This is a strong assumption and is not always satisfied. For example, in complex models where θ influences both x and y in intertwined ways such as 2-d Gaussian with full covariance matrix parameters. Only for generative models where y is generated given x and x is independently distributed, you can factor the joint distribution like you did above. Commented Jul 29 at 4:29
• Please see hayashi pg. 47 and statchurski pg. 123. There are numerous more resources online (including cross-validated) that assume independence of regression model parameter with the marginal distribution. Commented Jul 29 at 10:56
• Sure for regression as stated in your 2nd ref: For the following theoretical discussion, we suppose that the pairs $(x_n,y_n)$ are independent of each other and share a common (joint) density $p$, which is exactly what I wrote in my answer for this post. For regression problems we usually implicitly assume a common joint distribution over the random vector $(X,Y)$ which generates observed data $(x_n,y_n)$. So what else in my answer you're still not clear? I replied your question above about my answer and haven't heard from feedback yet. Commented Jul 29 at 22:17

For your assumed continuous response random variable in a regression model, your notation of the conditional pdf in your first conclusion is confused as it's mixing conditional with joint, the correct notation should be $$p_{Y|X}(y_i|x_i)$$. Also the observed response label $$Y_i$$ may be discrete as in the usual probabilistic logistic regression for classification problems where bear in mind you need to use pmf notation instead of pdf.

As for your second point it's certainly correct as we implicitly assume a fixed joint distribution $$p_{X,Y}$$ over the random vector $$(X,Y)$$ for your specific task (at least for your concerned inference stage without consideration of more advanced distribution shifts and other calibration prediction issues) whose marginal distribution over $$X$$ can be fixed too by integrating out $$Y$$ according to basic probability relation between joint and any of its marginal distributions, thus the conditional distribution function is fixed too for each random vector but their values $$p_{Y|X}(y_i|x_i)$$ may not be the same since $$(x_i,y_i)$$ are mostly likely different.

Finally by above fixed joint distribution $$p_{X,Y}$$ conclusion, each actually observed data vector $$(x_i,y_i)$$ is just a simple random sample (SRS) from the same joint distribution and by definition of SRS they're a sequence of independent, identically distributed (IID) random data points.

Addition after feedback from comment: The subtle part as for regression which seems to keep confusing OP is that in regression we don't compute the joint likelihood $$L((x_i,y_i);\theta)$$ at all where OP started their own answer, instead we compute $$L(y_i|x_i;\theta)$$ for each observed data point since regression treats $$x_i$$ as a constant, not a sample from a random variable $$X_i$$, for example in linear regression $$L(y_i|x_i;\theta)=\mathcal{N}(x_i^T\beta,\sigma)$$, where it's assumed $$y_i=x_i^T\beta+\epsilon_i$$, $$\epsilon_i \overset{\text{iid}} \sim \mathcal{N}(0,\sigma)$$.

• I do not get why the DGP for a regression model needs to be I.I.D. From my point 1. as long as the DGP is independent, the respective conditional distributions are independent see - stats.stackexchange.com/questions/644953/… . Having the conditional distribution functions identical for each random vector as well does not really have any implications on the DGP to be I.I.D. Commented Jul 27 at 11:39
• If the DGP for a regression model is only independent with possibly different conditional distributions for each data vector pair, it's equivalent to already have too many distribution shifts even in the training stage thus you're almost bound to fail to infer a moderately performant predictive model whose optimal case is just the identical conditional distribution. Such out-of-distribution (OOD) data is a common challenge in real world operations to have a well calibrated statistical model which is a field of research in itself. Commented Jul 28 at 4:22
• On the other hand, of course if the DGP is independent and each data pair is not identically distributed like in your real world econometric time series case, their different conditional distributions are independent which is consistent with your linked post. However, more importantly, such a single time series cannot be used for any autoregression (AR) since the assumption for a AR model is stationarity with IID noise term during each step which is inconsistent with your non-identical (conditional) distributions assumption. In summary your assumption is possible but not for any regression. Commented Jul 28 at 4:53