A quick sidenote: the Poisson distribution and the exponential distribution are two faces of the same coin. The Poisson distributions is a count process for events whereas the exponential distribution models time between those same events.
I know the Beta distribution is a convenient way to model the uncertainty in $p$ in a Binomial distribution because the Beta distribution is flexible (with the $\alpha$ and $\beta$ parameters) and exists between 0 and 1 (just like probabilities should be). But is there more to the story? Are the Beta distribution and Binomial distribution two faces of the same coin as well?