Is it better to select distributions based on theory, fit or something else?

Question

This is bordering on a philosophical question, but I am interested in how others with more experience think about distribution selection. In some cases it seems clear that theory might work best (mice tail lengths are probably normally distributed). In a lot of cases there is probably no theory to describe a set of data, so you just use something that fits what you have fairly well regardless of what it was originally developed to describe? I can imagine some of the pitfalls of going with one or the other of these, and then of course there seems to be the problem that maybe you should just use an empirical distribution if you really have no idea.

So I guess what I'm really asking: does someone have a coherent way of approaching/thinking about this problem? And are there any resources you can suggest that give a good treatment of this?

Answer 1

Definitely depends on what the data in question are and how much one knows or wishes to assume about them. As @whuber said in chat recently, "Where physical law is involved, you almost always can make reasonable guesses about an appropriate way to model the data." (I suspect this is truer of him than it is of me though! Also, I hope this isn't misapplied out of its original context...) In cases more like latent construct modeling in social sciences, it's often useful to focus on empirical distributions as a way of understanding the nuances of lesser-known phenomena. It's somewhat too easy to assume a normal distribution and dismiss misfit in the overall shape as negligible, and it's quite specious to dismiss outliers as erroneous without more justification than that they don't fit a theoretical distribution.

Of course, much of this behavior is motivated by the assumptions of analyses one wants to apply. Often the most interesting questions go far beyond description or classification of variables' distributions. This also influences the right answer for a given scenario; there may be reasons (e.g., power needs) to assume a normal distribution when it doesn't fit particularly well (nor misfit too badly), since nonparametric and otherwise robust methods aren't perfect either. Nonetheless, the risk of doing so habitually is forgetting to ask the interesting questions one can ask about a single variable's distribution.

For example, consider the relationship between wealth and happiness: a popular question people generally want to ask. It might be safe to assume wealth follows a gamma ^{(Salem & Mount, 1974)} or generalized beta ^{(Parker, 1999)} distribution, but is it really safe to assume happiness is normally distributed? Really, it shouldn't be necessary to assume this at all just to answer the original question, but people sometimes do, and then ignore potentially important issues like response bias and cultural differences. For instance, some cultures tend to give more or less extreme responses (see @chl's answer on Factor analysis of questionnaires composed of Likert items), and norms vary with regard to the open expression of positive and negative emotion ^{(Tucker, Ozer, Lyubomirsky, & Boehm, 2006)}. This may increase the importance of differences in empirical distributional characteristics like skewness and kurtosis. If I were comparing the relationship of wealth to subjective ratings of happiness in Russia, China, and the USA, I'd probably want to assess differences in central tendencies of happiness ratings. In doing so, I'd hesitate to assume normal distributions across each for the sake of a one-way ANOVA (even though it might be fairly robust to violations) when there's reason to expect a "fatter-tailed" distribution in China, a positively skewed distribution in Russia, and a negatively skewed distribution in the USA due to various culture-dependent norms and response biases. For the sake of a significance test (even though I'd probably prefer to just report effect sizes, honestly), I'd rather use a nonparametric method, and for the sake of actually understanding subjective happiness in each population individually, I'd rather describe the distribution empirically than try to categorize it as some simple theoretical distribution and ignore or gloss over any misfit. That's a waste of info IMO.

<sup>References
- Parker, S. C. (1999). The generalised beta as a model for the distribution of earnings. Economics Letters, 62(2), 197–200.
- Salem, A. B. Z., & Mount, T. D. (1974). A convenient descriptive model of income distribution: The gamma density. Econometrica, 42(6), 1115–1127.
- Tucker, K. L., Ozer, D. J., Lyubomirsky, S., & Boehm, J. K. (2006). Testing for measurement invariance in the satisfaction with life scale: A comparison of Russians and North Americans. Social Indicators Research, 78(2), 341–360. Retrieved from http://drsonja.net/wp-content/themes/drsonja/papers/TOLB2006.pdf.</sup>

Answer 2

mice tail lengths are probably normally distributed

I would doubt that. Normal distributions arise from many independent additive effects. Biological systems consist of many interacting feedback loops (inter-dependent multiplicative effects). Also there are often some states that are more stable than others (ie attractors). So some kind of long tailed or multimodal distribution would probably describe tail lengths. In fact, the normal distribution is probably a very poor default choice to describe anything biological and it's misuse is responsible for the many "outliers" reported in that literature. The prevalence of this distribution in nature is a myth and not just in the "perfect circles don't really exist" sense. However it does not follow that the mean and sd are useless as summary statistics. The problem arises when these are looked at only instead of the data rather than in addition.

Especially because I cannot readily determine when it might be better to "trust the data" (like this one funky right skewed data set I have, but n=160 which given the data doesn't seem like enough) and go with empirical, or fit it to a Beta distribution like a colleague of mine keeps insisting. I suspected he selected that only because it is bounded on [0,1]. It all just seems really ad hoc. Hopefully this clarifies my intent!

Fitting empirical distributions provides hints at the underlying process, which facilitates the development of theoretical distributions. Then the theoretical distribution is compared to the empirical distributions to test the evidence for the theory.

If your purpose is assessing the probability of certain outcomes based on the current evidence available and you have no reason to choose that particular distribution I guess I don't see how making additional assumptions could be helpful. Instead it seems to confuse matters.

However, if you are attempting to describe or summarize the data then it may make sense to fit the distribution.

Answer 3

In some cases it seems clear that theory might work best (mice tail lengths are probably normally distributed).

Tail lengths are certainly not normally distributed.

Normal distributions have a nonzero probability of taking negative values; tail lengths do not.

George Box's famed line, "all models are wrong, but some are useful" makes the point rather well. Cases where we might reasonably assert normality (rather than just approximate normality) are very rare indeed, almost creatures of legend, mirages occasionally almost glimpsed out of the corner of the eye.

In a lot of cases there is probably no theory to describe a set of data, so you just use something that fits what you have fairly well regardless of what it was originally developed to describe?

In cases where the quantities you're interested in are not especially sensitive to the choice (as long as the broad features of the distribution are consistent with what's known), then yes, you can just use something that fits fairly well.

In cases where there is a greater degree of sensitivity, 'just using something that fits' isn't sufficient on its own. We might use some approach that doesn't make particular assumptions (perhaps distribution free procedures, like permutation, bootstrapping or other resampling approaches, or robust procedures). Alternatively we might quantify the sensitivity to the distributional assumption, such as via simulation (indeed I think this is generally a good idea).

there seems to be the problem that maybe you should just use an empirical distribution if you really have no idea.

I wouldn't describe that as a problem - basing inference on empirical distributions certainly a legitimate approach suitable for many kinds of problems (permutation/randomization and bootstrapping are two examples).

does someone have a coherent way of approaching/thinking about this problem?

broadly, in a lot of cases, I tend to consider questions like:

1) What do I understand* about how means (or other location-type quantities) behave for data of this form?

*(whether from theory, or experience of this form of data, or expert advice, or if necessary, from the data itself, though that carries problems one must deal with)

2) What about spread (variance, IQR, etc) - how does it behave?

3) What about other distributional features (bounds, skewness, discreteness, etc)

4) What about dependence, heterogeneity of populations, tendency to occasionally very discrepant values, etc

This sort of consideration might guide a choice between a normal model, a GLM, some other model or some robust or distribution-free approach (such as bootstrapping or permutation/randomization approaches, including rank-based procedures)