Learning an efficient $f$(text) → Beta$(\alpha, \beta)$ Say we want to learn an efficient encoding of a function $f(\text{text})\rightarrow \text{Beta}(\alpha, \beta)$, where text is e.g.  an URL. We want to do this for millions of URLs.
That is, given a dataset made of millions of $\text{URLs}$ and $f(\text{URL})$'s, we we want to learn an efficient encoding $\hat{f}$ of $f$ that minimizes the error between $f(\text{URL})$ and $\hat{f}(\text{URL})$ (e.g. the KL divergence between the Betas).
I believe this is an example of learning embeddings (sparse representation learning). I am guessing variational autoencoders or DNNs could help here, but I am seeking advice in general.  What is the literature and/or state of the art in this topic? 
 A: Given a dataset of $\{(\text{URL}_n, (\alpha_n, \beta_n))\}_{n=1}^N$ our goal is to learn two embeddings $\alpha(\text{URL})$ and $\beta(\text{URL})$ that minimize
$$
\sum_{n=1}^N
D_{KL}(\text{Beta}(\alpha_n, \beta_n) \mid\mid \text{Beta}(\alpha(\text{URL}_n), \beta(\text{URL}_n))) \to \min_{\alpha, \beta}
$$
(Note that KL-divergence is not symmetric, and particular choice might be important. More on that later)
If we allowed arbitrary functions $\alpha$ and $\beta$, the best solution would be to just memorize all URLs and their parameters. However, that approach doesn't scale, so instead we'll assume we have a neural network g(URL), that outputs two numbers: $\alpha(\text{URL})$ and $\beta(\text{URL})$, and denote its set of parameters as $\Theta$.
$$
L(\theta) := \sum_{n=1}^N
D_{KL}(\text{Beta}(\alpha_n, \beta_n) \mid\mid \text{Beta}(\alpha_\Theta(\text{URL}_n), \beta_\Theta(\text{URL}_n))) \to \min_{_\Theta}
$$
Now all we have left is to rewrite this objective such that we can obtain a (stochastic) gradient, or automatically differentiate the objective:
(Note: I'll be using $\text{Beta}(x \mid \alpha, \beta)$ for density function at point $x$. I will not use cumulative density function, so that should not cause confusion)
$$
\begin{align*}
L(\Theta) &= 
\sum_{n=1}^N
\int \text{Beta}(x \mid \alpha_n, \beta_n) \log \frac{\text{Beta}(x \mid \alpha_n, \beta_n)}{\text{Beta}(x \mid \alpha_\Theta(\text{URL}_n), \beta_\Theta(\text{URL}_n))} dx \\
&= 
-\sum_{n=1}^N
\int \text{Beta}(x \mid \alpha_n, \beta_n) \log \text{Beta}(x \mid \alpha_\Theta(\text{URL}_n), \beta_\Theta(\text{URL}_n)) dx + \text{const}(\Theta) \\
&= 
-\sum_{n=1}^N
\mathbb{E}_{x \sim \text{Beta}(\alpha_n, \beta_n)} \log \text{Beta}(x \mid \alpha_\Theta(\text{URL}_n), \beta_\Theta(\text{URL}_n)) + \text{const}(\Theta) \\
&= 
-\sum_{n=1}^N
\mathbb{E}_{x \sim \text{Beta}(\alpha_n, \beta_n)} \left[ (\alpha_\Theta(\text{URL}_n)-1) \log x + (\beta_\Theta(\text{URL}_n)-1) \log(1-x) \\- \log \frac{\Gamma(\alpha_\Theta(\text{URL}_n)) \Gamma(\beta_\Theta(\text{URL}_n))}{\Gamma(\alpha_\Theta(\text{URL}_n)) \Gamma(\beta_\Theta(\text{URL}_n))} \right] + \text{const}(\Theta) \\
&= 
-\sum_{n=1}^N
\left[ \alpha_\Theta(\text{URL}_n) (\psi(\alpha_n) - \psi(\alpha_n + \beta_n)) + \beta_\Theta(\text{URL}_n) (\psi(\beta_n) - \psi(\alpha_n + \beta_n)) \\- \log \frac{\Gamma(\alpha_\Theta(\text{URL}_n)) \Gamma(\beta_\Theta(\text{URL}_n))}{\Gamma(\alpha_\Theta(\text{URL}_n)) \Gamma(\beta_\Theta(\text{URL}_n))} \right] + \text{const}(\Theta)
\end{align*}
$$
Where $\psi$ is a digamma function (see wikipedia on logarithms of Beta random variables).
You can write this objective in any framework that supports automatic differentiation (Theano, TensorFlow), and get a symbolic expression for the gradient for $\Theta$ (Provided your framework is able to differentiate Gamma function).
Of course, $N$ is too big for batch gradient descent, so you'd want to do minibatch optimization.
Choice of KL divergence: now let's get back to discussion of KL-divergence's asymmetry. Had I chosen another order, e.g. $D_{KL}(\text{neural net} \mid \mid \text{data})$, the whole inference would be totally different and much harder! Indeed, we'd have to take expectation w.r.t. our neural net distribution $\text{Beta}(\alpha_\Theta(\text{URL}), \beta_\Theta(\text{URL}))$, and differentiating it would require using score function and possibly some variance reduction technique.
You mentioned Variational Autoencoders, but they're crucially different in one very important aspect: they work with distributions that allow reparametrization trick to evade differentiating expectation's distribution. I know, VAE paper claims this transformation is possible for Beta / Dirichlet distributions, but in my research I figured it's likely just a mistake, and ratio of 2 scaled Gammas will not give you a Beta-distributed random variable. So, at the moment reparametrization trick for Beta / Dirichlet distributions seem to be unknown to the scientific community. There's logistic normal distribution, but it's unimodal and thus can't favor sparsity.
Choice of a neural network: So far I assumed $\alpha_\Theta$ and $\beta_\Theta$ are two output of some neural network with weights $\Theta$, but didn't discuss any concrete architectures. And in fact I don't have any specific recommendations, when it comes to deep learning, only experiments can help you. The most adequate architecture seems to be a many-to-one RNN, but you might want to experiment with other options as well.
