2
$\begingroup$

I have been working lately a lot on amortized variational inference. That is, doing variational inference using neural networks to approximate a variational distribution (such as in Kingma and Welling 2013 and Rezende and Mohammed 2014). I have read a lot on the matter and even implemented some of the basic models. I am now wondering why exogenous variables are never used to approximate the variational distribution. I think this would make sense in a lot of models. To give an example, if you wanted to do an economic model at country level of a specific variable (economic growth across time) you could use other country level variables to approximate the variational distribution better. Am I wrong? what am I missing?

PS. I know that, for example, the Conditional VAE (Kihyuk et.al. 2015) could count as using exogenous variables in amortized variational inference BUT you are forced to condition both in the inference and generative model...

$\endgroup$
  • $\begingroup$ Thank you for the answer and the reference. I'm sorry about the "out of nowhere" definition of exogenous, in economics (maybe in other sciences is a bit different) exogenous refers to variables that are not affected by other variables in the model. As I said, I already know the conditional generative models, I didn't know you could marginalize over them which might be something I should at. In any case, after thinking about thoroughly, it doesn't make sense for my model to not condition both on the inference and the generation model. $\endgroup$ – Sergio Jul 23 at 14:15
  • $\begingroup$ I've moved my comment to an answer. For marginalization, it depends on the model. If you have explicit decoder densities you can actually compute $\int p_\theta(x,y) dy$. Otherwise, you can use a prior $p(y)$ (either learned or fixed, e.g. uniform), when sampling (instead of specifying $y$). (For exogenous, I think we call them auxiliary variables in CS?Not sure if there's a commonly accepted term.) $\endgroup$ – user3658307 Jul 23 at 15:29
2
$\begingroup$

I'm not familiar with the term exogenous but it is common to have additional variables (often, "labels"; for instance, the category $y$ of an image $x$). Then $y\sim q_\psi(x),\, z \sim q_\phi(x,y)$ for inference and $x\sim p_\theta(x,y)$ for generation. You don't have to use $y$ in the generative model (you could even marginalize it out), but usually the whole point of these particular models is to allow controlled generation of novel samples via $y$. Maybe see Learning Disentangled Representations with Semi-Supervised Deep Generative Models by Narayanaswamy et al?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.