Is “joint probability” assumption necessary for regression purposes?

Question

I state beforehand that my question may sound odd and captious (and maybe it is).

In regression theory basically we assume that explanatory variables and independent variable are joined togheter generally speaking with an unknown joint distribution.

All that we're looking for is a linear approximation (the best one) of the conditional expectation (CEF) $E[Y\mid\bf{x}]$ with $\bf{x}$ being the vector of regressors.

Clearly if we know the exact underlying distribution we're done (e.g. if $(Y,\bf{x})$ is MVN the conditional expectation is linear and it has a straighforward expression written on books) .

In OLS theory we look for a linear approximation of the CEF collecting a bunch of data $(y_1,{\bf{x}}_1 ),\ldots,(y_n,{\bf{x}}_n )$ that we suppose drawn from an identical underlying population (this is the standard assumption that the observations are identically distributed), in order for the coefficient $\beta=E[{\bf{xx}}^t]^{-1}E[{\bf{x}}y]$ to make sense.

---

My problem is how we can be sure that the joint underlying distribution exists, it's one and somehow fixed?

I mean, let's suppose that I want to make some prediction about $children's \; height$ using as regressors $parent's \; height $ (recalling the famous Francis Galton paper), who can assure us these assumptions?

If there's another variable which acts in an "odd manner", modifying the joint shape over time, is that analogous to an omitted bias problem?

Answer 1

All models are wrong, but some are more useful than others. -- George E P Box

You have made a great observation! The short answer to "how we can be sure that the joint underlying distribution exists, it's one and somehow fixed?" is: we can't be sure! The longer answer is: whatever distributional assumptions you make will be wrong, and you can learn a lot by understanding how they are wrong and by trying to make them less wrong. For example, you can plot residuals against time, and see if there is an important trend.

If you find evidence of an important trend, then you can investigate modifying your model's assumptions to be less wrong. You could investigate simple modifications, such as just including time as an ordinary independent variable in an ordinary linear regression, or more complicated modifications, like abandoning linear regression in favor of an explicit time series model.

Through this process of criticizing your model and finding appropriate modifications to model important trends in your data, you will, well, understand the important trends in your data :) This journey of model-based exploration is usually more insightful than p-values you can read off of the destination model.

Answer 2

Often it is assumed that $Y_i$ is a random variable with expected value $a+bx_i$ and variance $\sigma^2>0$ and covariances $\operatorname{cov}(Y_i,Y_j)=0$ for $i\ne j.$ The parameters $a,b,\sigma$ are to be estimated based on the data and $x_i$ are observed rather than to be estimated.

Notice that the above attributes no randomness to $x_i.$

Sometimes $x_i$ are fixed by design, so they have no randomness. In that case, if a new random sample of $n$ observations $Y_i,\,i=1,\ldots,n$ is taken, independently of the first sample of $n$ observations, the $Y_i$ change but the $x_i$ do not.

However, sometimes in practice a new sample would alter both $x_i$ and $Y_i$. In that case, often the same assumptions as in the first paragraph above are made. This may be justified by the fact that what is of interest is the conditional distribution of the $Y_i$ given the $x_i.$