# Is it better to select distributions based on theory, fit or something else?

This is bordering on a philosophical question, but I am interested in how others with more experience think about distribution selection. In some cases it seems clear that theory might work best (mice tail lengths are probably normally distributed). In a lot of cases there is probably no theory to describe a set of data, so you just use something that fits what you have fairly well regardless of what it was originally developed to describe? I can imagine some of the pitfalls of going with one or the other of these, and then of course there seems to be the problem that maybe you should just use an empirical distribution if you really have no idea.

So I guess what I'm really asking: does someone have a coherent way of approaching/thinking about this problem? And are there any resources you can suggest that give a good treatment of this?

• It depends fundamentally on why one is fitting or assuming a distribution and what it is intended to represent. We field many questions on this site where it appears people feel they have to fit a distribution to data or derived quantities (like regression residuals) when in fact the exercise is pointless (or worse, deceptive) as far as solving the statistical problems they really have is concerned. Could you perhaps clarify the kinds of cases you have in mind? – whuber Mar 28 '14 at 16:36
• Hi Whuber, thanks for the comment. Since I have started working a little on probablistic risk assessment, I'm required to fit all of my data to distributions and it made me curious about having a more consistent view on how distribution selection is done. So I guess to clarify, I'm only really interested in the times when you should be using a distribution, and how to go about it correctly. Like I said, some cases have been easy from theory, other times I'm using an empirical distribution because it seems best, but my decision making is more haphazard than I'd like. – HFBrowning Mar 28 '14 at 16:43
• That's an interesting can of worms, because what you are really doing (somewhat abstractly) is trying to propagate sampling uncertainty through a calculation. The reason for looking at the procedure from this high level is that it reveals a fundamental mistake that is often made: by replacing the data by distributions, one fails to include uncertainty in the estimated distribution parameters. Accounting for this is called "second order" PRA by some practitioners. I would like to suggest you narrow your question to focus on these issues rather than asking about distribution fitting in general. – whuber Mar 28 '14 at 16:48
• The package I am using for my PRA is a 2nd order monte carlo (mc2d package in R), so I am assigning my distributions either as "uncertainty", "variability" or both. So hopefully I am accounting for that problem as far as I can. However, my original intent for this question was to gain a higher level view, and I brought up risk assessment simply to give context for why I am interested. And perhaps there is no better way than "sometimes you do this, sometimes you do it that way" but I was hoping someone had suggestions :) Especially because I cannot readily determine when it might be better - – HFBrowning Mar 28 '14 at 16:57
• This is definitely the right place for your post. Are you saying you are having trouble making the edits? Incidentally, I am curious about how your procedures quantify the uncertainty in using the empirical distribution. It, too, comes with sampling variability (which can be profound in the tails, which often matter the most in risk assessments), even though you have not explicitly estimated any parameters. – whuber Mar 28 '14 at 19:00

Definitely depends on what the data in question are and how much one knows or wishes to assume about them. As @whuber said in chat recently, "Where physical law is involved, you almost always can make reasonable guesses about an appropriate way to model the data." (I suspect this is truer of him than it is of me though! Also, I hope this isn't misapplied out of its original context...) In cases more like latent construct modeling in social sciences, it's often useful to focus on empirical distributions as a way of understanding the nuances of lesser-known phenomena. It's somewhat too easy to assume a normal distribution and dismiss misfit in the overall shape as negligible, and it's quite specious to dismiss outliers as erroneous without more justification than that they don't fit a theoretical distribution.

Of course, much of this behavior is motivated by the assumptions of analyses one wants to apply. Often the most interesting questions go far beyond description or classification of variables' distributions. This also influences the right answer for a given scenario; there may be reasons (e.g., needs) to assume a normal distribution when it doesn't fit particularly well (nor misfit too badly), since and otherwise methods aren't perfect either. Nonetheless, the risk of doing so habitually is forgetting to ask the interesting questions one can ask about a single variable's distribution.

For example, consider the relationship between wealth and happiness: a popular question people generally want to ask. It might be safe to assume wealth follows a gamma (Salem & Mount, 1974) or generalized beta (Parker, 1999) distribution, but is it really safe to assume happiness is normally distributed? Really, it shouldn't be necessary to assume this at all just to answer the original question, but people sometimes do, and then ignore potentially important issues like response bias and cultural differences. For instance, some cultures tend to give more or less extreme responses (see @chl's answer on Factor analysis of questionnaires composed of Likert items), and norms vary with regard to the open expression of positive and negative emotion (Tucker, Ozer, Lyubomirsky, & Boehm, 2006). This may increase the importance of differences in empirical distributional characteristics like skewness and kurtosis. If I were comparing the relationship of wealth to subjective ratings of happiness in Russia, China, and the USA, I'd probably want to assess differences in central tendencies of happiness ratings. In doing so, I'd hesitate to assume normal distributions across each for the sake of a one-way ANOVA (even though it might be fairly robust to violations) when there's reason to expect a "fatter-tailed" distribution in China, a positively skewed distribution in Russia, and a negatively skewed distribution in the USA due to various culture-dependent norms and response biases. For the sake of a significance test (even though I'd probably prefer to just report effect sizes, honestly), I'd rather use a nonparametric method, and for the sake of actually understanding subjective happiness in each population individually, I'd rather describe the distribution empirically than try to categorize it as some simple theoretical distribution and ignore or gloss over any misfit. That's a waste of info IMO.

References
- Parker, S. C. (1999). The generalised beta as a model for the distribution of earnings. Economics Letters, 62(2), 197–200.
- Salem, A. B. Z., & Mount, T. D. (1974). A convenient descriptive model of income distribution: The gamma density. Econometrica, 42(6), 1115–1127.
- Tucker, K. L., Ozer, D. J., Lyubomirsky, S., & Boehm, J. K. (2006). Testing for measurement invariance in the satisfaction with life scale: A comparison of Russians and North Americans. Social Indicators Research, 78(2), 341–360. Retrieved from http://drsonja.net/wp-content/themes/drsonja/papers/TOLB2006.pdf.

• Thanks for your answer, Nick. I found the example especially helpful. – HFBrowning Mar 31 '14 at 15:41

mice tail lengths are probably normally distributed

I would doubt that. Normal distributions arise from many independent additive effects. Biological systems consist of many interacting feedback loops (inter-dependent multiplicative effects). Also there are often some states that are more stable than others (ie attractors). So some kind of long tailed or multimodal distribution would probably describe tail lengths. In fact, the normal distribution is probably a very poor default choice to describe anything biological and it's misuse is responsible for the many "outliers" reported in that literature. The prevalence of this distribution in nature is a myth and not just in the "perfect circles don't really exist" sense. However it does not follow that the mean and sd are useless as summary statistics. The problem arises when these are looked at only instead of the data rather than in addition.

Especially because I cannot readily determine when it might be better to "trust the data" (like this one funky right skewed data set I have, but n=160 which given the data doesn't seem like enough) and go with empirical, or fit it to a Beta distribution like a colleague of mine keeps insisting. I suspected he selected that only because it is bounded on [0,1]. It all just seems really ad hoc. Hopefully this clarifies my intent!

Fitting empirical distributions provides hints at the underlying process, which facilitates the development of theoretical distributions. Then the theoretical distribution is compared to the empirical distributions to test the evidence for the theory.

If your purpose is assessing the probability of certain outcomes based on the current evidence available and you have no reason to choose that particular distribution I guess I don't see how making additional assumptions could be helpful. Instead it seems to confuse matters.

However, if you are attempting to describe or summarize the data then it may make sense to fit the distribution.

• Even though I can only accept one answer, I wanted to thank you for pointing out how normal distributions actually arise. It forced me to think more carefully about what it means for something to be based on theory. – HFBrowning Mar 31 '14 at 18:39

In some cases it seems clear that theory might work best (mice tail lengths are probably normally distributed).

Tail lengths are certainly not normally distributed.

Normal distributions have a nonzero probability of taking negative values; tail lengths do not.

George Box's famed line, "all models are wrong, but some are useful" makes the point rather well. Cases where we might reasonably assert normality (rather than just approximate normality) are very rare indeed, almost creatures of legend, mirages occasionally almost glimpsed out of the corner of the eye.

In a lot of cases there is probably no theory to describe a set of data, so you just use something that fits what you have fairly well regardless of what it was originally developed to describe?

In cases where the quantities you're interested in are not especially sensitive to the choice (as long as the broad features of the distribution are consistent with what's known), then yes, you can just use something that fits fairly well.

In cases where there is a greater degree of sensitivity, 'just using something that fits' isn't sufficient on its own. We might use some approach that doesn't make particular assumptions (perhaps distribution free procedures, like permutation, bootstrapping or other resampling approaches, or robust procedures). Alternatively we might quantify the sensitivity to the distributional assumption, such as via simulation (indeed I think this is generally a good idea).

there seems to be the problem that maybe you should just use an empirical distribution if you really have no idea.

I wouldn't describe that as a problem - basing inference on empirical distributions certainly a legitimate approach suitable for many kinds of problems (permutation/randomization and bootstrapping are two examples).

broadly, in a lot of cases, I tend to consider questions like:

1) What do I understand* about how means (or other location-type quantities) behave for data of this form?

*(whether from theory, or experience of this form of data, or expert advice, or if necessary, from the data itself, though that carries problems one must deal with)

2) What about spread (variance, IQR, etc) - how does it behave?

3) What about other distributional features (bounds, skewness, discreteness, etc)

4) What about dependence, heterogeneity of populations, tendency to occasionally very discrepant values, etc

This sort of consideration might guide a choice between a normal model, a GLM, some other model or some robust or distribution-free approach (such as bootstrapping or permutation/randomization approaches, including rank-based procedures)