Is there any Generative Model which can be used for Regression problems?

Question

I've been researching Generative Models recently, and Probabilistic Graphical Models.

Every time I read about Generative Models, I see they're trying to predict $P(x,y)$ or equivalently $P(x|y)$ and $P(y)$, and I understand this. But every example uses classification problems and makes a comparison to discriminative models, which learn either $P(y|x)$ ($\propto P(x|y)P(y) = P(x,y)$ by Bayes' Rule) or even simpler just a map from $x \rightarrow y$.

In regression problems we generally just try to find that relationship $x \rightarrow y$. But shouldn't it in principle be possible to find a probability distribution over continuous output values too? Shouldn't it in principle be possible to model the whole of a continuously-valued problem with probability distributions internally? You would need to keep $P(x, y)$ as a function rather than as a table of discrete values, but I see no reason that should be impossible.

Are there learning models that do that sort of probabilistic reasoning for regression problems? Are there any that could be considered Generative which can handle the regression case?

Whenever I Google it, I just see results for Logistic Regression, which has long seemed like a misnomer to me, because LR is for classification.

Answer 1

Of course generative models work for continuous variables. Especially, you can extend any regression model with a prior distribution on x and a specific noise model on y to yield a generative model for that situation.

As an example for simple linear 1D regression x->y you could additionally assume that x comes from a normal distribution $N(\mu_x,\sigma^2_x)$ and there is noise on the result such that $y = \beta x + \alpha + \epsilon , \epsilon \sim N(0,\sigma^2_\epsilon)$.

Then you have a perfectly valid generative model:

\begin{eqnarray} P(x,y) &=& P(x) P(y|x)\\ &=& N(x|\mu_x,\sigma^2_x) N(y|\beta x+\alpha,\sigma^2_\epsilon) \end{eqnarray}

which will actually simplify to be a 2D Gaussian model for x and y.

Similarly virtually all regressions can be turned into a generative model if you are willing to additionally assume a prior on $x$ and an explicit noise model.

Answer 2

I am also thinking about this problem. What kind of model would you get if you make the output of Gaussian discriminant model continuous, namely:

\begin{align*} \boldsymbol y_{(i)} &\sim \mathcal N (\mu_1, \sigma^2) \\ [X_{(i)}, \boldsymbol y_{(i)}] &\sim \mathcal N(\boldsymbol\mu_2, \Sigma) \end{align*}

Note that the second distribution have $\boldsymbol y_{(i)}$ joined together with different features of $X_{(i)}$ somehow makes it generate different distributions with respect to $X_{(i)}$, albeit I believe that would be just linear changes in $\boldsymbol\mu$ and $\Sigma$. If you want more complicated variations, you can have $\boldsymbol\mu$ and $\Sigma$ modeled by some other distributions and make them depend on $\boldsymbol y_{(i)}$

Then maximize the joint log probability, to make the expression prettier, extract the last variable of $\boldsymbol\mu_2$: $\boldsymbol\mu_2^T := [\boldsymbol\mu_2^T, \mu_3]$:

\begin{align*} \max_{\boldsymbol \mu_1, \boldsymbol \mu_2,\mu_3} \sum_{i=1}^m \ln \int_{-\infty}^\infty \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(\frac{\pmb y_{(i)} - \mu_1}{2\sigma^2} \right) \cdot \frac{1}{\sqrt{(2\pi)^n |\Sigma|}} \quad \exp\left( \begin{bmatrix} (X_{(i)} - \boldsymbol\mu_2)^T & \boldsymbol y_{(i)} -\mu3 \end{bmatrix} \Sigma^{-1} \begin{bmatrix} X_{(i)} - \boldsymbol\mu_2 \\ \boldsymbol y_{(i)} - \mu3 \end{bmatrix} \right) \text{d}\boldsymbol y_{(i)} \end{align*}

I will try to work this out if I have time. I don't even know how to do that integral.

I guess if you want make the output continuous you have to do an integral over the entire $\mathbb{R}$, and if that doesn't converge or doesn't have an analytical solution, you model won't work/

Answer 3

Decided to take another look at this. Managed to find that GMMs are considered generative, and it is possible to do regression with them, though this is so uncommon that sklearn doesn't have it.

There must be other examples. I'm still unclear on exactly how these examples work or what others may exist.