26
$\begingroup$

The answers (definitions) defined on Wikipedia are arguably a bit cryptic to those unfamiliar with higher mathematics/statistics.

In mathematical terms, a statistical model is usually thought of as a pair ($S, \mathcal{P}$), where $S$ is the set of possible observations, i.e. the sample space, and $\mathcal{P}$ is a set of probability distributions on $S$.

In probability and statistics, a probability distribution assigns a probability to each measurable subset of the possible outcomes of a random experiment, survey, or procedure of statistical inference. Examples are found whose sample space is non-numerical, where the distribution would be a categorical distribution.

I am a high school student very interested in this field as a hobby and am currently struggling with the differences between what is a statistical model and a probability distribution

My current, and very rudimentary, understanding is this:

  • statistical models are mathematical attempts to approximate measured distributions

  • probability distributions are measured descriptions from experiments that assigns probabilities to each possible outcome of a random event

confusion is further compounded by the tendency in literature to see the words "distribution" and "model" used interchangeably - or at least in very similar situations (e.g. binomial distribution vs binomial model)

Can someone verify/correct my definitions, and perhaps offer a more formalized (albeit still in terms of simple english) approach to these concepts?

$\endgroup$
  • 1
    $\begingroup$ Bottom line: there is absolutely no difference between a statistical model and a probability distribution. Every statistical model describes a probability distribution and vice versa. Don't let them confuse you with long texts. $\endgroup$ – Cowboy Trader May 2 '16 at 18:38
  • 3
    $\begingroup$ @Cagdas According to the definition cited in the question, there is a difference: a statistical model is a particular organized collection of probability distributions. When only one probability distribution is in evidence, then we are no longer doing statistics at all, because the aim of statistical analysis has been achieved: we know the distribution! $\endgroup$ – whuber May 2 '16 at 20:17
  • 2
    $\begingroup$ @cagdas Wikipedia keeps company with the best texts. I fully agree with it. $\endgroup$ – whuber May 3 '16 at 10:42
  • 4
    $\begingroup$ @CagdasOzgenc, why not present some evidence to substantiate your sharp and definite claims. Proof by authority is rarely (if ever) acceptable. Without eivdence it is difficult (if not impossible) to have a productive discussion; unsubstantiated claims are rarely much more than noise. $\endgroup$ – Richard Hardy May 3 '16 at 10:52
  • 2
    $\begingroup$ @RichardHardy The question asked "layman terms", and look at the answers he got. Excuse me but I hate to see students suffer just because somebody decides to show off. The answer is as simple as 2+2=4, and I really don't think it requires a 20 page authoritative reference. $\endgroup$ – Cowboy Trader May 3 '16 at 14:09
25
$\begingroup$

Probability distribution is a mathematical function that describes a random variable. A little bit more precisely, it is a function that assigns probabilities to numbers and it's output has to agree with axioms of probability.

Statistical model is an abstract, idealized description of some phenomenon in mathematical terms using probability distributions. Quoting Wasserman (2013):

A statistical model $\mathfrak{F}$ is a set of distributions (or densities or regression functions). A parametric model is a set $\mathfrak{F}$ that can be parameterized by a finite number of parameters. [...]

In general, a parametric model takes the form

$$ \mathfrak{F} = \{ f (x; \theta) : \theta \in \Theta \} $$

where $\theta$ is an unknown parameter (or vector of parameters) that can take values in the parameter space $\Theta$. If $\theta$ is a vector but we are only interested in one component of $\theta$, we call the remaining parameters nuisance parameters. A nonparametric model is a set $\mathfrak{F}$ that cannot be parameterized by a finite number of parameters.

In many cases we use distributions as models (you can check this example). You can use binomial distribution as a model of counts of heads in series of coin throws. In such case we assume that this distribution describes, in simplified way, the actual outcomes. This does not mean that this is an only way on how you can describe such phenomenon, neither that binomial distribution is something that can be used only for this purpose. Model can use one or more distributions, while Bayesian models specify also prior distributions.

More formally this is discussed by McCullaugh (2002):

According to currently accepted theories [Cox and Hinkley (1974), Chapter 1; Lehmann (1983), Chapter 1; Barndorff-Nielsen and Cox (1994), Section 1.1; Bernardo and Smith (1994), Chapter 4] a statistical model is a set of probability distributions on the sample space $\mathcal{S}$. A parameterized statistical model is a parameter $\Theta$ set together with a function $P : \Theta \rightarrow \mathcal{P} (\mathcal{S})$, which assigns to each parameter point $\mathcal{\theta \in \Theta}$ a probability distribution $P \theta$ on $\mathcal{S}$. Here $\mathcal{P}(\mathcal{S})$ is the set of all probability distributions on $\mathcal{S}$. In much of the following, it is important to distinguish between the model as a function $ P : \Theta \rightarrow \mathcal{P} (\mathcal{S}) $, and the associated set of distributions $P\Theta \subset \mathcal{P} (\mathcal{S})$.

So statistical models use probability distributions to describe data in their terms. Parametric models are also described in terms of finite set of parameters.

This does not mean that all statistical methods need probability distributions. For example, linear regression is often described in terms of normality assumption, but in fact it is pretty robust to departures from normality and we need assumption about normality of errors for confidence intervals and hypothesis testing. So for regression to work we don't need such assumption, but to have fully specified statistical model we need to describe it in terms of random variables, so we need probability distributions. I write about this because you can often hear people saying that they used regression model for their data -- in most such cases they rather mean that they describe data in terms of linear relation between target values and predictors using some parameters, than insisting on conditional normality.


McCullagh, P. (2002). What is a statistical model? Annals of statistics, 1225-1267.

Wasserman, L. (2013). All of statistics: a concise course in statistical inference. Springer.

$\endgroup$
  • 4
    $\begingroup$ @J.C.Leitão that is why I added the notice ;) The classical OLS is only about fitting the line. Normality assumptions are only about the noise while the core idea is that we are modelling E(y) as a linear function of X. Normality is needed for confidence intervals and testing, but regression is about fitting the line and errors are of lesser importance. (Loosely speaking.) $\endgroup$ – Tim May 2 '16 at 18:17
  • $\begingroup$ Thank you for your answer. Could you provide 2 concise definitions to summarize? (also I dont understand the last line In much of the following, it is important to distinguish between the model as a function and the associated set of distributions) Are you just making a comment of the inherent ambiguity between the two meanings sharing the same term model or am I missing something? $\endgroup$ – AlanSTACK May 3 '16 at 4:42
  • $\begingroup$ @Alan two concise definition are provided in the first two paragraphs, while more rigorous one in the quote and references - could you clarify what is unclear? As about the last line of quote: it basically says that model is defined in terms of probability distributions and parameters and it is good to remember that is has those two aspects, sometimes it is good to distinguish them. I recommend the quoted paper for rigorous discussion (it is freely available under the link). $\endgroup$ – Tim May 3 '16 at 9:31
8
$\begingroup$

Think of $\mathcal{S}$ as a set of tickets. You can write stuff on a ticket. Usually a ticket starts out with the name of some real-world person or object that it "represents" or "models." There's lots of blank space on each ticket for writing other things.

You can make as many copies of each ticket as you want. A probability model $\mathbb{P}$ for this real-world population or process consists of making one or more copies of every ticket, mixing them up, and putting them in a box. If you--the analyst--can establish that the process of drawing one ticket randomly from this box emulates all the important behavior of what you are studying, then you can learn much about the world by thinking about this box. Because some tickets may be more numerous in the box than others, they may have difference chances of being drawn. Probability theory studies these chances.

When numbers are written on the tickets (in a consistent way), they give rise to (probability) distributions. A probability distribution merely describes the proportion of tickets in a box whose numbers lie within any given interval.

Because we usually don't know exactly how the world behaves, we have to imagine different boxes in which the tickets appear with different relative frequencies. The set of these boxes is $\mathcal{P}$. We view the world as being adequately described by the behavior of one of the boxes in $\mathcal{P}$. It is your objective to make reasonable guesses as to which box it is, based on what you see on the tickets you have pulled out of it.


As an example (which is practical and realistic, not a textbook toy), suppose you are studying the rate $y$ of a chemical reaction as it varies with temperature. Suppose that the theory of chemistry predicts that within the range of temperatures between $0$ and $100$ degrees, the rate is proportional to the temperature.

You plan to study this reaction at both $0$ and $100$ degrees, making several observations at each temperature. You therefore make up a very, very large number of boxes. You are going to fill each box with tickets. There is a rate constant written on each one. All the tickets in any given box have the same rate constant written on them. Different boxes use different rate constants.

Using the rate constant written on any ticket, you also write down the rate at $0$ and the rate at $100$ degrees: call these $y_0$ and $y_{100}$. But this is not yet enough for a good model. Chemists also know that no substance is pure, no quantity is exactly measured, and other forms of observational variability occur. To model these "errors," you make very, very many copies of your tickets. On each copy you change the values of $y_0$ and $y_{100}$. On most of them you change them only a little. On a very few, you might change them a lot. You write down as many changed values as you plan to observe at each temperature. These observations represent possible observable outcomes of your experiment. Into the box go each such set of these tickets: it is a probability model for what you might observe for a given rate constant.

What you do observe is modeled by drawing a ticket from that box and reading only the observations written there. You don't get to see the underlying (true) values of $y_0$ or $y_{100}$. You don't get to read the (true) rate constant. Those aren't afforded by your experiment.

Every statistical model must make some assumptions about the tickets in these (hypothetical) boxes. For instance, we hope that when you modified the values of the $y_0$ and $y_{100}$, you did so without consistently increasing or consistently decreasing either one (as a whole, within the box): that would be a form of systematic bias.

Because the observations written on each ticket are numbers, they give rise to probability distributions. The assumptions made about the boxes typically are phrased in terms of properties of those distributions, such as whether they must average out to zero, be symmetric, have a "bell curve" shape, are uncorrelated, or whatever.


That's really all there is to it. Much in the way that a primitive twelve-tone scale gave rise to all of Western classical music, a collection of ticket-containing boxes is a simple concept that can be used in extremely rich and complex ways. It can model just about anything, ranging from a coin flip to a library of videos, databases of Website interactions, quantum mechanical ensembles, and anything else that can be observed and recorded.

$\endgroup$
3
$\begingroup$

The definition of a distribution as assigning probabilities to each possible event works for discrete distribution, but becomes trickier for continuous distributions, where e.g. any number on the real line could be the outcome. Very often when talking about distributions, we think of them as having fixed parameters such as a binomial distribution having two parameters: firstly, the number of observations and secondly a probability $\pi$ of a single observation being an event.

Typical parametric statistical models describe how the parameter(s) of a distribution depend on certain things such as factors (a variable that has discrete values) and covariates (continuous variables). For example, if in a normal distribution you assume that the mean can be described by some fixed number (an "intercept") and some number (a "regression coefficient") times the value of a covariate, you obtain a linear regression model with a normally distributed error term. For a binomial distribution, one commonly used model ("logistic regression") is to assume that the logit of the probability $\pi$ of an event ($\pi/(1-\pi)$) can be described by a regression equation such as $\text{intercept}+\beta_1 \text{covariate}_1+\ldots$. Similarly, for a Poisson distribution a common model is to assume this for the logarithm of the rate parameter ("Poisson regression").

$\endgroup$
  • 2
    $\begingroup$ Yes, but... Model is not only about parameters but also can be about structure of the problem (e.g. probabilstic model that resembles the assumed data generating process); there are also non-parametric models. $\endgroup$ – Tim May 2 '16 at 14:24
2
$\begingroup$

A probability distribution gives all the information about how a random quantity fluctuates. In practice we usually do not have the full probability distribution of our quantity of interest. We may know or assume something about it without knowing or assuming that we know everything about it. For example, we might assume that some quantity is normally distributed but know nothing about the mean and variance. Then we have a collection of candidates for the distribution to choose from; in our example, it is all possible normal distributions. This collection of distributions forms a statistical model. We use it by gathering data and then restricting our class of candidates so that all the remaining candidates are consistent with the data in some appropriate sense.

$\endgroup$
2
$\begingroup$

A model is specified by a PDF, but it is not a PDF.

Probability distribution (PDF) is a function that assigns probabilities to numbers and its output has to agree with axioms of probability, like Tim explained.

A model is fully defined by a probability distribution, but it is more than that. In the coin tossing example, our model could be "coin is fair" + "each throw is independent". This model is specified by a PDF that is a binomial with p=0.5.

However, one could imagine a model where the throws are not independent, in which case it is no longer described by the binomial PDF. Still, the model is specified by the joint distribution (a PDF) of all events $P(x_1, x_2, x_3, ...)$. The point being, formally, a model is always specified by the joint distribution over events.

One distinction between the model and the PDF is that a model can be interpreted as a statistical hypothesis. For example, in coin tossing, we can consider the model where the coin is fair (p=0.5), and that each throw is independent (binomial), and say that this is our hypothesis, which we want to test against a competing hypothesis.

You can also have competing models (e.g. we don't know $p$ and we want to compute which $p$ is the best fit). It does not make sense to speak of competing PDFs because they are just a mathematical object.

$\endgroup$
  • $\begingroup$ Can you elaborate on your last sentence? That seems to be a major part of nonparametric statistics, to me. $\endgroup$ – Ian May 2 '16 at 18:20
  • $\begingroup$ I always interpreted non-parametric models as less restrictive on the PDF of x_i, but that still require a PDF for the statistics they use. E.g. Kendal rank correlation assumes normality to compute the p-value. But it could be that there is a counter example. I would be interested. $\endgroup$ – Jorge Leitão May 2 '16 at 20:11
  • $\begingroup$ I just don't understand what you mean when you say "it does not make sense to speak of competing PDFs". This is exactly what we are really doing, even in parametric statistics: we have a bunch of PDFs that we think might be valid for the problem, we take some data, and we conclude from the data that some subset of our PDFs is better. Then we quantify what we mean by "better". (Also, in the elementary context, you really should not use "PDF" for everything. In the distributional sense this ultimately works out, but this is quite sophisticated machinery...) $\endgroup$ – Ian May 2 '16 at 20:22
  • $\begingroup$ A model is specified by a PDF I disagree. A model might be specified by multiple PDF as well. And a model might be specified by no PDF: think of something like a SVM or regression tree. $\endgroup$ – Ricardo Cruz May 4 '16 at 8:43
2
$\begingroup$

You ask a very important question, Alan, and have received some fine answers above. I would like to offer a simpler answer, and also indicate an additional dimension to the distinction that the above answers have not addressed. For simplicity, everything I'll say here relates to parametric statistical models.

First of all, you may find the idea of a family helpful for connecting your question with things you've learned in high school. (I am surprised that this word has not yet appeared on this page!) You long ago learned about the quadratic family of curves, $y = a x^2 + b x + c$. You can think of a parametric statistical model in the same way, as a family of distributions. You have probably done lab experiments in chemistry or physics classes, where you collected data and plotted them in order to identify parameters from a simple family of models like $y = m x + b$ or $F = -k x$. At the highest level, estimating the parameters of a statistical model very much resembles the process of finding the slope $m$ and intercept $b$, or finding the spring constant $k$. As you continue to study mathematics, you will see 'families' of various sorts of entities pop up everywhere.

So, my brief Answer #1 to your question is: a statistical model is a family of distributions.

The further point I wanted to make relates to the qualifier, statistical. As Judea Pearl points out in his "golden rule of causal analysis" [1,p350],

No causal claim can be established by a purely statistical method, be it propensity scores, regression, stratification, or any other distribution-based design.

(For present purposes, I would invite you to read "statistical" in place of "distribution-based," and "model" in place of "design.") What Pearl is keen to convey is that our models of causal effects in the world (think $F=-kx$, for example!) necessarily embody more than purely statistical ideas. Thus, taking your question as titled---i.e., without the qualification statistical attached to model---a full answer requires the further admonition that models generally incorporate causal ideas that lie inherently outside the province of statistics, i.e. of statements about probability distributions.

Thus, my Answer #2 to your question is: models usually embody causal ideas that cannot be expressed in purely distributional terms.


[1]: Pearl, Judea. Causality: Models, Reasoning and Inference. 2nd edition. Cambridge, U.K. ; New York: Cambridge University Press, 2009. Link to §11.3.5, including cited p. 351.

$\endgroup$
  • $\begingroup$ Forgive my ignorance, but what do you mean with the word causal? Is there some more nuanced meaning to it or does it simply refer to the notion of causality and relationships bound between causes and effects? Thank you for your answer, btw. $\endgroup$ – AlanSTACK May 4 '16 at 6:34
  • $\begingroup$ Causal knowledge involves the effects of interventions. If you have causal knowledge, then you know how some system will respond to an action you make. (Cf. the common refrain, "association is not causation.") One way to appreciate how causal knowledge lies beyond the province of mere statistics is to consider the Hooke's Law example I cited above. Depending on how a spring is used (e.g., in a fish scale vs spring-loaded toy gun), the $F$ might cause the $x$ or vice versa. Yet $F=-kx$ is ambivalent to the causality here (because $=$ is a symmetric relation). $\endgroup$ – David C. Norris May 4 '16 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.