In layman's terms, what is the difference between a model and a distribution?

Question

The answers (definitions) defined on Wikipedia are arguably a bit cryptic to those unfamiliar with higher mathematics/statistics.

In mathematical terms, a statistical model is usually thought of as a pair ($S, \mathcal{P}$), where $S$ is the set of possible observations, i.e. the sample space, and $\mathcal{P}$ is a set of probability distributions on $S$.

In probability and statistics, a probability distribution assigns a probability to each measurable subset of the possible outcomes of a random experiment, survey, or procedure of statistical inference. Examples are found whose sample space is non-numerical, where the distribution would be a categorical distribution.

I am a high school student very interested in this field as a hobby and am currently struggling with the differences between what is a statistical model and a probability distribution

My current, and very rudimentary, understanding is this:

• statistical models are mathematical attempts to approximate measured distributions

• probability distributions are measured descriptions from experiments that assigns probabilities to each possible outcome of a random event

confusion is further compounded by the tendency in literature to see the words "distribution" and "model" used interchangeably - or at least in very similar situations (e.g. binomial distribution vs binomial model)

Can someone verify/correct my definitions, and perhaps offer a more formalized (albeit still in terms of simple english) approach to these concepts?

Answer 1

Probability distribution is a mathematical function that describes a random variable. A little bit more precisely, it is a function that assigns probabilities to numbers and it's output has to agree with axioms of probability.

Statistical model is an abstract, idealized description of some phenomenon in mathematical terms using probability distributions. Quoting Wasserman (2013):

A statistical model $\mathfrak{F}$ is a set of distributions (or densities or regression functions). A parametric model is a set $\mathfrak{F}$ that can be parameterized by a finite number of parameters. [...]

In general, a parametric model takes the form

$$ \mathfrak{F} = \{ f (x; \theta) : \theta \in \Theta \} $$

where $\theta$ is an unknown parameter (or vector of parameters) that can take values in the parameter space $\Theta$. If $\theta$ is a vector but we are only interested in one component of $\theta$, we call the remaining parameters nuisance parameters. A nonparametric model is a set $\mathfrak{F}$ that cannot be parameterized by a finite number of parameters.

In many cases we use distributions as models (you can check this example). You can use binomial distribution as a model of counts of heads in series of coin throws. In such case we assume that this distribution describes, in simplified way, the actual outcomes. This does not mean that this is an only way on how you can describe such phenomenon, neither that binomial distribution is something that can be used only for this purpose. Model can use one or more distributions, while Bayesian models specify also prior distributions.

More formally this is discussed by McCullaugh (2002):

According to currently accepted theories [Cox and Hinkley (1974), Chapter 1; Lehmann (1983), Chapter 1; Barndorff-Nielsen and Cox (1994), Section 1.1; Bernardo and Smith (1994), Chapter 4] a statistical model is a set of probability distributions on the sample space $\mathcal{S}$. A parameterized statistical model is a parameter $\Theta$ set together with a function $P : \Theta \rightarrow \mathcal{P} (\mathcal{S})$, which assigns to each parameter point $\mathcal{\theta \in \Theta}$ a probability distribution $P \theta$ on $\mathcal{S}$. Here $\mathcal{P}(\mathcal{S})$ is the set of all probability distributions on $\mathcal{S}$. In much of the following, it is important to distinguish between the model as a function $ P : \Theta \rightarrow \mathcal{P} (\mathcal{S}) $, and the associated set of distributions $P\Theta \subset \mathcal{P} (\mathcal{S})$.

So statistical models use probability distributions to describe data in their terms. Parametric models are also described in terms of finite set of parameters.

This does not mean that all statistical methods need probability distributions. For example, linear regression is often described in terms of normality assumption, but in fact it is pretty robust to departures from normality and we need assumption about normality of errors for confidence intervals and hypothesis testing. So for regression to work we don't need such assumption, but to have fully specified statistical model we need to describe it in terms of random variables, so we need probability distributions. I write about this because you can often hear people saying that they used regression model for their data -- in most such cases they rather mean that they describe data in terms of linear relation between target values and predictors using some parameters, than insisting on conditional normality.

---

McCullagh, P. (2002). What is a statistical model? Annals of statistics, 1225-1267.

Wasserman, L. (2013). All of statistics: a concise course in statistical inference. Springer.

Answer 2

Think of $\mathcal{S}$ as a set of tickets. You can write stuff on a ticket. Usually a ticket starts out with the name of some real-world person or object that it "represents" or "models." There's lots of blank space on each ticket for writing other things.

You can make as many copies of each ticket as you want. A probability model $\mathbb{P}$ for this real-world population or process consists of making one or more copies of every ticket, mixing them up, and putting them in a box. If you--the analyst--can establish that the process of drawing one ticket randomly from this box emulates all the important behavior of what you are studying, then you can learn much about the world by thinking about this box. Because some tickets may be more numerous in the box than others, they may have difference chances of being drawn. Probability theory studies these chances.

When numbers are written on the tickets (in a consistent way), they give rise to (probability) distributions. A probability distribution merely describes the proportion of tickets in a box whose numbers lie within any given interval.

Because we usually don't know exactly how the world behaves, we have to imagine different boxes in which the tickets appear with different relative frequencies. The set of these boxes is $\mathcal{P}$. We view the world as being adequately described by the behavior of one of the boxes in $\mathcal{P}$. It is your objective to make reasonable guesses as to which box it is, based on what you see on the tickets you have pulled out of it.

---

As an example (which is practical and realistic, not a textbook toy), suppose you are studying the rate $y$ of a chemical reaction as it varies with temperature. Suppose that the theory of chemistry predicts that within the range of temperatures between $0$ and $100$ degrees, the rate is proportional to the temperature.

You plan to study this reaction at both $0$ and $100$ degrees, making several observations at each temperature. You therefore make up a very, very large number of boxes. You are going to fill each box with tickets. There is a rate constant written on each one. All the tickets in any given box have the same rate constant written on them. Different boxes use different rate constants.

Using the rate constant written on any ticket, you also write down the rate at $0$ and the rate at $100$ degrees: call these $y_0$ and $y_{100}$. But this is not yet enough for a good model. Chemists also know that no substance is pure, no quantity is exactly measured, and other forms of observational variability occur. To model these "errors," you make very, very many copies of your tickets. On each copy you change the values of $y_0$ and $y_{100}$. On most of them you change them only a little. On a very few, you might change them a lot. You write down as many changed values as you plan to observe at each temperature. These observations represent possible observable outcomes of your experiment. Into the box go each such set of these tickets: it is a probability model for what you might observe for a given rate constant.

What you do observe is modeled by drawing a ticket from that box and reading only the observations written there. You don't get to see the underlying (true) values of $y_0$ or $y_{100}$. You don't get to read the (true) rate constant. Those aren't afforded by your experiment.

Every statistical model must make some assumptions about the tickets in these (hypothetical) boxes. For instance, we hope that when you modified the values of the $y_0$ and $y_{100}$, you did so without consistently increasing or consistently decreasing either one (as a whole, within the box): that would be a form of systematic bias.

Because the observations written on each ticket are numbers, they give rise to probability distributions. The assumptions made about the boxes typically are phrased in terms of properties of those distributions, such as whether they must average out to zero, be symmetric, have a "bell curve" shape, are uncorrelated, or whatever.

---

That's really all there is to it. Much in the way that a primitive twelve-tone scale gave rise to all of Western classical music, a collection of ticket-containing boxes is a simple concept that can be used in extremely rich and complex ways. It can model just about anything, ranging from a coin flip to a library of videos, databases of Website interactions, quantum mechanical ensembles, and anything else that can be observed and recorded.

Answer 3

The definition of a distribution as assigning probabilities to each possible event works for discrete distribution, but becomes trickier for continuous distributions, where e.g. any number on the real line could be the outcome. Very often when talking about distributions, we think of them as having fixed parameters such as a binomial distribution having two parameters: firstly, the number of observations and secondly a probability $\pi$ of a single observation being an event.

Typical parametric statistical models describe how the parameter(s) of a distribution depend on certain things such as factors (a variable that has discrete values) and covariates (continuous variables). For example, if in a normal distribution you assume that the mean can be described by some fixed number (an "intercept") and some number (a "regression coefficient") times the value of a covariate, you obtain a linear regression model with a normally distributed error term. For a binomial distribution, one commonly used model ("logistic regression") is to assume that the logit of the probability $\pi$ of an event ($\pi/(1-\pi)$) can be described by a regression equation such as $\text{intercept}+\beta_1 \text{covariate}_1+\ldots$. Similarly, for a Poisson distribution a common model is to assume this for the logarithm of the rate parameter ("Poisson regression").

Answer 4

A probability distribution gives all the information about how a random quantity fluctuates. In practice we usually do not have the full probability distribution of our quantity of interest. We may know or assume something about it without knowing or assuming that we know everything about it. For example, we might assume that some quantity is normally distributed but know nothing about the mean and variance. Then we have a collection of candidates for the distribution to choose from; in our example, it is all possible normal distributions. This collection of distributions forms a statistical model. We use it by gathering data and then restricting our class of candidates so that all the remaining candidates are consistent with the data in some appropriate sense.

Answer 5

You ask a very important question, Alan, and have received some fine answers above. I would like to offer a simpler answer, and also indicate an additional dimension to the distinction that the above answers have not addressed. For simplicity, everything I'll say here relates to parametric statistical models.

First of all, you may find the idea of a family helpful for connecting your question with things you've learned in high school. (I am surprised that this word has not yet appeared on this page!) You long ago learned about the quadratic family of curves, $y = a x^2 + b x + c$. You can think of a parametric statistical model in the same way, as a family of distributions. You have probably done lab experiments in chemistry or physics classes, where you collected data and plotted them in order to identify parameters from a simple family of models like $y = m x + b$ or $F = -k x$. At the highest level, estimating the parameters of a statistical model very much resembles the process of finding the slope $m$ and intercept $b$, or finding the spring constant $k$. As you continue to study mathematics, you will see 'families' of various sorts of entities pop up everywhere.

So, my brief Answer #1 to your question is: a statistical model is a family of distributions.

The further point I wanted to make relates to the qualifier, statistical. As Judea Pearl points out in his "golden rule of causal analysis" [1,p350],

No causal claim can be established by a purely statistical method, be it propensity scores, regression, stratification, or any other distribution-based design.

(For present purposes, I would invite you to read "statistical" in place of "distribution-based," and "model" in place of "design.") What Pearl is keen to convey is that our models of causal effects in the world (think $F=-kx$, for example!) necessarily embody more than purely statistical ideas. Thus, taking your question as titled---i.e., without the qualification statistical attached to model---a full answer requires the further admonition that models generally incorporate causal ideas that lie inherently outside the province of statistics, i.e. of statements about probability distributions.

Thus, my Answer #2 to your question is: models usually embody causal ideas that cannot be expressed in purely distributional terms.

---

[1]: Pearl, Judea. Causality: Models, Reasoning and Inference. 2nd edition. Cambridge, U.K. ; New York: Cambridge University Press, 2009. Link to §11.3.5, including cited p. 351.

Answer 6

A model is specified by a PDF, but it is not a PDF.

Probability distribution (PDF) is a function that assigns probabilities to numbers and its output has to agree with axioms of probability, like Tim explained.

A model is fully defined by a probability distribution, but it is more than that. In the coin tossing example, our model could be "coin is fair" + "each throw is independent". This model is specified by a PDF that is a binomial with p=0.5.

However, one could imagine a model where the throws are not independent, in which case it is no longer described by the binomial PDF. Still, the model is specified by the joint distribution (a PDF) of all events $P(x_1, x_2, x_3, ...)$. The point being, formally, a model is always specified by the joint distribution over events.

One distinction between the model and the PDF is that a model can be interpreted as a statistical hypothesis. For example, in coin tossing, we can consider the model where the coin is fair (p=0.5), and that each throw is independent (binomial), and say that this is our hypothesis, which we want to test against a competing hypothesis.

You can also have competing models (e.g. we don't know $p$ and we want to compute which $p$ is the best fit). It does not make sense to speak of competing PDFs because they are just a mathematical object.