Can a neural net approximate any conditional density asymptotically?

Question

Assume that the conditional density of $ y \vert x $ is a Beta distribution for all values of x. Can a Beta distribution with parameters computed by a neural net, i.e. Beta($\hat{\alpha}$, $\hat{\beta}$) where $\begin{bmatrix} \hat{\alpha} & \hat{\beta} \end{bmatrix} = f(x; \theta)$ where $ f(x;\theta) $ is a neural net parameterised by $ \theta $, approximate $ y \vert x $ asymptotically? Is the following proposition true if $ f(x; \theta) $ is a neural net that can approximate any continuous function: $$ \exists \theta \ s.t. \ [ \ p(y|x) = Beta(f(x; \theta)), \ \forall{x,y} ] $$ ?

Answer 1

If I understand you correctly, you have the $y_i$ values that conditionally on $x_i$ follow the beta distribution and the relationship can be described as

$$ (\alpha_i, \beta_i) = f(x; \theta) \\ y_i \sim \mathsf{Beta}(\alpha_i, \beta_i) $$

where $f(\cdot; \theta)$ is an unknown function to be estimated. Can neural networks do this? Sure. There's a beta regression that does this but uses a simpler model to approximate the function $f$. Just train it by maximizing the likelihood function defined in terms of beta distribution for the $y_i$ targets and use whatever neural network architecture to approximate $f$. One thing that you could do to simplify the model is to re-parametrize beta distribution in terms of location and precision, as beta regression does.

If you would like to learn the distribution of $y_i$'s, you would need to switch to Bayesian ground, use something like KL divergence and switch from standard neural network framework to probabilistic programming one, like PyMC3, Pyro, TensorFlow Probability, etc. So the how is a longer discussion.