Choosing among PDFs This is a pretty broad question. I just learned that two random variables can have the same moments but different PDFs. Take $\mathbb{E}[X_i] = \mu$ and $\mathbb{Var}[X_i] = \sigma^2$. Since there are many PDFs which could have these same moments (and even higher order moments), what factors should one consider when choosing among PDFs to model $X_i$? I know that in Bayesian analysis, conjugacy is important for computational reasons, but what are some more theoretical reasons to choose one PDF over another? What properties of the data might an experimenter consider when choosing a model?
 A: Indeed, even all (positive integer) moments may in some situations be insufficient to distinguish two distributions. Generally, the first few moments don't 'pin down' a distribution very well, which suggests not using moments for that purpose.
Models (in various senses) for how the variables arise can sometimes help.
For example, taking the case of counts, there are various "count data models" - 
a) from a sequence of Bernoulli trials (with the assumptions that entails), such as the number of successes in $n$ trials (binomial), the number of trials (or the number of failures) to the $k$-th success (negative binomial); 
b) if sampling is from a finite population without replacement we instead get hypergeometric and negative hypergeometric
(not to mention the Poisson and various other models)
In these situations, as a first step, consideration of the situation itself may be sufficient to choose a reasonable model
With continuous variables, similar considerations may come into it -- with times for example, we might consider exponential or gamma distributions (via the connection to times is the Poisson process); or we might more easily choose some model for speed(/rate) rather than time. In any case we would often expect times to be skewed due to the lower bound, so that narrows things down a bit.
Similarly, in the case of sizes of things, various models, such as 'broken stick' models lead to particular choices of distribution.
Or when values are extremes, asymptotic results form extreme value theory may reduce consideration to a few candidates as a first attempt.
Consideration should also be given to known bounds; if your data are the proportion of time spent in some activity, the result is (almost surely) continuous and bounded (it must lie between 0 and 1, for example, though the lower bound will often be the more important of the two). So choice should take that into account. In addition, smoothness, ability to deal with the kind of nonconstant variance that tends to arise, or convenience/interpretability may be factors in the choice.
Often there are several different convenient distributions with more-or-less similar properties that may all be reasonably adequate for some purpose. It's often the case with data where spread is roughly proportional to the mean and conditional distributions are right skew that both gamma and lognormal models are convenient. If interest focuses on say predicting the mean, the analyses (and level of effort) may be fairly similar -- which means that while in those circumstances it may be more difficult to pick one, the choice may also be less critical.
