1
$\begingroup$

I am trying to see if there any Statistical Models that (better) "Exploit" Distributional Knowledge of the Predictor Variables.

For example, I feel that is a common misconception (e.g Where does the misconception that Y must be normally distributed come from?) regarding the requirement of the Normal Distribution in Regression Modelling. As I understand, the response variable does not need to be Normally Distributed. However:

  • The response variable needs to be "conditionally normal" on the covariates (i.e. E( Y| X=x )
  • The residuals needs to be normally distributed (i.e. (Y-truth - Y-predicted) ~ Normal Distribution)

Especially when considering GLM's, these requirements are quite convenient as they relax the distributional assumptions of the variables.

My question relates to the following: Suppose we happen to have information about the distributions of the covariate variables (e.g. suppose we have enough data and when we plot the histograms and run MLE tests, the data corresponds well to common probability distributions) - are there any statistical models that are able to "exploit" this information for the purpose of "enriching the quality" of the statistical models?

It seems to me that regression models do not require the covariates to have certain distributions (e.g. if we want to predict "age" based on "height and weight", the columns in our data corresponding to the "height" and "weight" variables do not need to be normally distributed) - however, at the same time, if we happened to know the distributions of "height" and "weight", it seems like regression models would not be able to make use of this information. If they could, is it possible that a potentially "better" statistical model could be made by "exploiting" this information?

I spent some time thinking about ways that statistical models can "exploit" distributional information, and came up with the following examples:

  • Suppose the response variable and the covariate variables in the data all have normal distributions, and we believe that we can model the joint probability distribution of the response variable and all covariates through a multivariate normal distribution, i.e. P(Y, X1, X2,..Xn)~ MVN (mu, sigma). Doing so, we have now effectively "exploited" the distributional information about the covariates (the regression model would have effectively ignored this information). In practice, we can now predict different values of Y conditional on X, e.g. P(Y| X = x)~ MVN (mu*, sigma*), effectively performing the same task as a regression model, but with the added benefit of potentially enriching our model by better exploiting available information on the covariates.

  • A similar approach can also be used even if each variable has is believed to have come from a different probability distribution. Copula models can create a joint probability distribution using the Cumulative Probability Distributions of each variable, thus allowing for each variable to have a fundamentally different probability distribution (e.g. height ~ Normal, Weight ~ Gamma, etc.). Random samples from the conditional distribution of the Copula model with respect to the response variable and some observed values of the covariates can be generated as well, also performing the same task a regression model P(Y| X = x) - but with the added benefit of potentially enriching our model by better exploiting available information on the covariates.

Can someone please comment on the above? Is what I have described above correct and relevant? In general, are there any Statistical Models that (better) "Exploit" Distributional Knowledge of the Predictor Variables?

Thanks!

$\endgroup$
6
  • 2
    $\begingroup$ Neither one of the two bullet points at the top of the question are true. There is no distributional requirement at all for regression analysis. All likelihood-based approaches "exploit" distributional information, the distributional information is where the likelihood function comes from. If you are estimating the parameters $\theta$ of a joint distribution $f(X,Y;\theta)$, then likelihood-based approaches would seem to do exactly what you are wondering about. $\endgroup$
    – jbowman
    Commented May 14, 2022 at 19:49
  • $\begingroup$ Thank you for your reply! I thought in regression analysis we assume that the conditional probabilities of y given the covariates has a normal distribution? $\endgroup$
    – stats_noob
    Commented May 14, 2022 at 20:06
  • 1
    $\begingroup$ What could you gain? If the conditional distribution of $Y \mid X=x$ have common parameters with the marginal distribution of $X$ you would gain something, but that seems rather special ... see stats.stackexchange.com/questions/144826/… and references there! $\endgroup$ Commented May 14, 2022 at 20:20
  • 1
    $\begingroup$ No, we don't assume that the conditional probabilities of $y$ given the covariates have a normal distribution. If we do make that assumption, we can make statements about the finite-sample distribution of the parameter estimates and the OLS estimates are also the MLE, which are why you sometimes see people making that assumption, but it is in no way necessary for pretty much anything else. $\endgroup$
    – jbowman
    Commented May 14, 2022 at 20:41
  • $\begingroup$ Research Errors in Variables (EIV) models for examples of models that use distributional assumptions about the predictors. But "exploit" sounds a little misleading, almost as if standard regression models somehow are inferior in neglecting to accommodate such information. The fact is that (most) people who are not "exploiting" such information are interested in results for which it is irrelevant. $\endgroup$
    – whuber
    Commented May 14, 2022 at 21:22

1 Answer 1

2
+25
$\begingroup$

In regression models, by definition, we are interested in the conditional distribution of the response variable $Y$ given the observed predictors (covariates) $X$. Namely, if the joint distribution can be parametrized as

$$ f(x,y|\theta,\psi) = f(y|x,\theta)f(x|\psi)$$

then only the parameters $\theta$ are relevant for the prediction. As pointed out by kjetil b halvorsen in the comments, you can only gain additional information form this joint distribution if the marginal distribution $f(x|\psi)$ has common parameters with the conditional distribution $f(y|x,\theta)$. But this is not usually the case: consider for example the bivariate normal that you mention in your question, which has

$$Y|X=x \sim \mathcal N \left( \mu_Y + \frac{\sigma_Y}{\sigma_X}\rho(x-\mu_X), (1-\rho^2)\sigma_Y^2\right) $$

$$ X \sim \mathcal N (\mu_X, \sigma_X^2) $$

Where the joint distribution is parametrized using the 5 parameters $\mu_X,\mu_Y,\sigma_X^2,\sigma_Y^2,\rho$. We can however use a different parametrization:

$$Y|X=x \sim \mathcal N \left( \alpha + \beta x, \hat \sigma^2\right) $$

Where we still have 5 parameters $\alpha,\beta,\hat\sigma^2,\mu_X,\sigma_X^2$, but now the conditional distribution of $Y$ only depends on 3 of them and the marginal distribution of $X$ depends on the remaining 2 (note that the original parametrization can be recovered from the new one and vice versa). It is only the 3 parameters $\alpha,\beta,\hat\sigma^2$ that are relevant for the prediction, so the marginal distribution of $X$ carries no additional information that we can exploit.

It is possible, in principle, to construct models that don't factorize in this manner, namely where

$$ f(x,y|\theta) = f(y|x,\theta)f(x|\theta)$$

Which implies that the marginal distribution of $X$ by itself already contains some information about the distribution of $Y$ (via the parameter $\theta$). This however might not be referred to as a "regression" model at all but simply as an estimation problem where one wishes to estimate the parameter $\theta$ based on data $x,y$ coming from a distribution $f(x,y|\theta)$.

So basically, the term "regression" will probably be mostly used for models that have the above factorization property, and otherwise they will just be described as general parameter estimation problems.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.