# Can a neural net approximate any conditional density asymptotically?

Assume that the conditional density of $$y \vert x$$ is a Beta distribution for all values of x. Can a Beta distribution with parameters computed by a neural net, i.e. Beta($$\hat{\alpha}$$, $$\hat{\beta}$$) where $$\begin{bmatrix} \hat{\alpha} & \hat{\beta} \end{bmatrix} = f(x; \theta)$$ where $$f(x;\theta)$$ is a neural net parameterised by $$\theta$$, approximate $$y \vert x$$ asymptotically? Is the following proposition true if $$f(x; \theta)$$ is a neural net that can approximate any continuous function: $$\exists \theta \ s.t. \ [ \ p(y|x) = Beta(f(x; \theta)), \ \forall{x,y} ]$$ ?

If I understand you correctly, you have the $$y_i$$ values that conditionally on $$x_i$$ follow the beta distribution and the relationship can be described as

$$(\alpha_i, \beta_i) = f(x; \theta) \\ y_i \sim \mathsf{Beta}(\alpha_i, \beta_i)$$

where $$f(\cdot; \theta)$$ is an unknown function to be estimated. Can neural networks do this? Sure. There's a beta regression that does this but uses a simpler model to approximate the function $$f$$. Just train it by maximizing the likelihood function defined in terms of beta distribution for the $$y_i$$ targets and use whatever neural network architecture to approximate $$f$$. One thing that you could do to simplify the model is to re-parametrize beta distribution in terms of location and precision, as beta regression does.

If you would like to learn the distribution of $$y_i$$'s, you would need to switch to Bayesian ground, use something like KL divergence and switch from standard neural network framework to probabilistic programming one, like PyMC3, Pyro, TensorFlow Probability, etc. So the how is a longer discussion.

• @HaziqMuhammad if you want to build a mathematical model then you need to describe the relationship as some kind of function. The function can be complicated and unknown, still neural network can learn such function from the data, the same as it learns all the other complicated and unknown functions.
– Tim
Jul 30 at 15:32
• AFAICS Tim's reference to beta regression is correct. You can add essentially any noise model you like to a neural network, most applications stick to Gaussian or Bernoulli, but there is nothing to stop you using more complicated models. For example P.M. Williams used a hybrid Bernoulli/Gamma network for modelling rainfall (I've also implemented it, it works really well) proceedings.neurips.cc/paper/1997/file/… Jul 30 at 15:34
• @HaziqMuhammad I don't understand what you are trying to say.
– Tim
Jul 30 at 16:16
• @HaziqMuhammad you use a neural network to approximate the unknown functional relationship between $x$ and $y$. The relationship itself is not a neural network, to the same extent as MNIST images were not created by the neural network, but are just scans of hand-written digits, where we can approximate the distribution of this data using a neural network.
– Tim
Jul 30 at 16:31
• @HaziqMuhammad you seem to be missing the approximation part, they are not equal. NN can approximate a function.
– Tim
Jul 30 at 17:21