I've been researching Generative Models recently, and Probabilistic Graphical Models.

Every time I read about Generative Models, I see they're trying to predict $P(x,y)$ or equivalently $P(x|y)$ and $P(y)$, and I understand this. But every example uses classification problems and makes a comparison to discriminative models, which learn either $P(y|x)$ ($\propto P(x|y)P(y) = P(x,y)$ by Bayes' Rule) or even simpler just a map from $x \rightarrow y$.

In regression problems we generally just try to find that relationship $x \rightarrow y$. But shouldn't it in principle be possible to find a probability distribution over continuous output values too? Shouldn't it in principle be possible to model the whole of a continuously-valued problem with probability distributions internally? You would need to keep $P(x, y)$ as a function rather than as a table of discrete values, but I see no reason that should be impossible.

Are there learning models that do that sort of probabilistic reasoning for regression problems? Are there any that could be considered Generative which can handle the regression case?

Whenever I Google it, I just see results for Logistic Regression, which has long seemed like a misnomer to me, because LR is for classification.

  • 2
    $\begingroup$ I guess any maximum likelihood regression model is generative: so eg linear regression under (eg) normal noise: $y ~ \alpha x + \epsilon$ $\endgroup$
    – seanv507
    Sep 24, 2018 at 19:56

3 Answers 3


Of course generative models work for continuous variables. Especially, you can extend any regression model with a prior distribution on x and a specific noise model on y to yield a generative model for that situation.

As an example for simple linear 1D regression x->y you could additionally assume that x comes from a normal distribution $N(\mu_x,\sigma^2_x)$ and there is noise on the result such that $y = \beta x + \alpha + \epsilon , \epsilon \sim N(0,\sigma^2_\epsilon)$.

Then you have a perfectly valid generative model:

\begin{eqnarray} P(x,y) &=& P(x) P(y|x)\\ &=& N(x|\mu_x,\sigma^2_x) N(y|\beta x+\alpha,\sigma^2_\epsilon) \end{eqnarray}

which will actually simplify to be a 2D Gaussian model for x and y.

Similarly virtually all regressions can be turned into a generative model if you are willing to additionally assume a prior on $x$ and an explicit noise model.

  • $\begingroup$ Of course! lol. I appreciate the simple concrete example. I'd still like to see an example in a larger Bayes Net or something to get a firmer intuition. $\endgroup$ Feb 20, 2020 at 18:27
  • $\begingroup$ A relatively simple extension to have some larger Bayesnet could be a hierarchical model where you assume that there is distribution over alpha and beta with parameters you want to estimate and you have groups of data for which you assume the same alpha and beta hold like locations or subjects $\endgroup$
    – Xenon
    Mar 2, 2020 at 19:20
  • $\begingroup$ should there be a comma between the parameters of the $P(x)$ Normal distribution? also, I was wondering what does the '$\mid$' mean in that notation. Thanks! $\endgroup$
    – synack
    Jul 9, 2020 at 7:38
  • $\begingroup$ yes there should be a , the | here just separates the variables this is a distribution over from the parameters. For normal distributions you often see a ; instead. Here I used the | for consistency with the conditional probabilities before it. $\endgroup$
    – Xenon
    Jul 10, 2020 at 8:24

I am also thinking about this problem. What kind of model would you get if you make the output of Gaussian discriminant model continuous, namely:

\begin{align*} \boldsymbol y_{(i)} &\sim \mathcal N (\mu_1, \sigma^2) \\ [X_{(i)}, \boldsymbol y_{(i)}] &\sim \mathcal N(\boldsymbol\mu_2, \Sigma) \end{align*}

Note that the second distribution have $\boldsymbol y_{(i)}$ joined together with different features of $X_{(i)}$ somehow makes it generate different distributions with respect to $X_{(i)}$, albeit I believe that would be just linear changes in $\boldsymbol\mu$ and $\Sigma$. If you want more complicated variations, you can have $\boldsymbol\mu$ and $\Sigma$ modeled by some other distributions and make them depend on $\boldsymbol y_{(i)}$

Then maximize the joint log probability, to make the expression prettier, extract the last variable of $\boldsymbol\mu_2$: $\boldsymbol\mu_2^T := [\boldsymbol\mu_2^T, \mu_3]$:

\begin{align*} \max_{\boldsymbol \mu_1, \boldsymbol \mu_2,\mu_3} \sum_{i=1}^m \ln \int_{-\infty}^\infty \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(\frac{\pmb y_{(i)} - \mu_1}{2\sigma^2} \right) \cdot \frac{1}{\sqrt{(2\pi)^n |\Sigma|}} \quad \exp\left( \begin{bmatrix} (X_{(i)} - \boldsymbol\mu_2)^T & \boldsymbol y_{(i)} -\mu3 \end{bmatrix} \Sigma^{-1} \begin{bmatrix} X_{(i)} - \boldsymbol\mu_2 \\ \boldsymbol y_{(i)} - \mu3 \end{bmatrix} \right) \text{d}\boldsymbol y_{(i)} \end{align*}

I will try to work this out if I have time. I don't even know how to do that integral.

I guess if you want make the output continuous you have to do an integral over the entire $\mathbb{R}$, and if that doesn't converge or doesn't have an analytical solution, you model won't work/


Decided to take another look at this. Managed to find that GMMs are considered generative, and it is possible to do regression with them, though this is so uncommon that sklearn doesn't have it.

There must be other examples. I'm still unclear on exactly how these examples work or what others may exist.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.