Can probability distributions be used as an alternative for regression models? Suppose you have 3 variables: height, weight and salary. Can you first attempt to fit a 3 dimensional probability distribution to this data - then, if someone gives you a height and weight measurement, could you use the probability distribution to estimate the probability of observing a range of salaries for a fixed value of weight and height?
This would in effect produce a conditional probability distribution for this specific combination and you can take the expected value of this distribution  to predict the most likely salary?
Here was my thought: once the 3 dimensional distribution was fit to this data, for a height = 180 cm and weight = 100 kg:
Probabilith (Salary = $10,000 | height = 180 cm, weight = 100 kg)  = 0.2
Probabilith (Salary = $10,500 | height = 180 cm, weight = 100 kg)  = 0.1
Probabilith (Salary = $9,000 | height = 180 cm, weight = 100 kg)  = 0.01
Etc.
Evaluating many such values of salary combinations, you can derive a posterior distribution. There would be no need for beta regression coefficients in this approach.
Is this correct?
Or this is a flawed idea and is it better to just fit a regression model to this problem?
Thanks!
 A: You can, but not without consequences.

*

*Linear regression is a pretty flexible model in terms of your ability to define the functional relationship between the features and the dependent variable. If you use multivariate distribution, you are limited by what kind of relationships between variables are possible under the distribution.

*For fitting the distribution you need to make more assumptions, for example, if you choose multivariate normal distribution you assume that all the variables follow normal distribution vs only $Y$ (conditionally) as in linear regression.

*For fitting the distribution you may need much more data. Think of discrete distribution: in conditional distribution case (regression) you need only enough data to observe relations of the other variables with $Y$, with the joint distribution you need data for all the combinations of all the levels of all the variables.

It is easier to focus only on the conditional distribution and conditional expectation, as we do with linear regression.
A: I am fairly sure it's valid to treat regression model as a joint probability distribution. Let $s,h,w$ be salary, height, weight, then a linear regression
$$
  s = \beta_0 + \beta_1h + \beta_2w + \epsilon \\
  \epsilon \sim N(0, \sigma^2)
$$
where $\beta_0, \beta_1, \beta_2$ are regression coefficients posits the distribution
$$
  s \sim N(\beta_0 + \beta_1h + \beta_2w, \sigma^2)
$$
The mean salary for a specific combination of weight and height is exactly what is being modelled in a linear regression, and this is also exactly what a conditional expectation of salary for a given weight and height would aim to provide in your approach.
Of course the accuracy of this prediction depends on the normality of the error term $\epsilon$; if this assumption is violated then predictions will not be accurate.
A: Yes, you definitely can. Moreover, it does not have to be any linear kind of regression. In the most general case, the following method will allow you to effectively turn any parametric or even non-parametric probability density estimation into a regressor for the desired output (i.e., the 3rd variable in your case, which is salary).
The following method is from the book:
Deep Learning, by Goodfellow, Bengio, and Courville, 2016 (page 103 or 104, from section 5.1.3), https://www.deeplearningbook.org/contents/ml.html.
Note: Despite the name of the book, the following method is completely general, and suitable beyond deep learning or any other neural network, or even other machine learning methods, for that matter.
The method:

*

*Assume you've modeled the probability distribution over the input vector $\textbf{v}$ as $p(\textbf{v})$ by any parametric or non-parametric probability density estimation technique. In your example: $\textbf{v}$ = [height, weight, salary] $\in R^3$.


*Now, you decide to estimate one component $y$ of the vector $\textbf{v}$ from the remaining "input" components $\textbf{x}$, i.e., estimate salary from height and weight. Let's denote: $\textbf{v} = (\textbf{x}, y)$, where $y$ is the desired component to be estimated, and $\textbf{x}$ are the remaining "input" components.


*Using the definition of conditional probability, the estimation of the probability of $y$ given the other components $\textbf{x}$ is:
$$p(y | \textbf{x}) = \frac{p(\textbf{x}, y)}{p(\textbf{x})} = \frac{p(\textbf{v})}{\sum_{y'}{p(\textbf{x}, y')}}$$
where: $$p(\textbf{x}) = \sum_{y'}{p(\textbf{x}, y')}$$ by the law of total probability, over quantized (discretized) values $y'$ of the component $y$. But you can use un-quantized continuous values using a suitable 1D integration technique of your choice:
$$p(\textbf{x}) = \int_{y'}{p(\textbf{x}, y')}dy'$$ by the law of total probability for continuous values $y'$ of the component $y$.
Observe that instead of a specific point-estimate of $y$ you obtain a posterior probability density estimation $p(y | \textbf{x})$ of $y$ given the $\textbf{x}$ inputs. If you only need a point-estimate, you can simply choose for example: $$\hat{y} = argmax_{y'} p(y | \textbf{x})$$ where $\hat{y}$ is the maximum aposteriori point-estimate.
