What's the difference between probability and statistics?

What's the difference between probability and statistics, and why are they studied together?

-

The short answer to this I've heard from Persi Diaconis is the following: the problems considered by probability and statistics are inverse to each other. In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens. In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.

-
So statistics observes what happens in the physical world, theorizes about the underlying process, and then having found the process, uses it in the sense of probability to predict what will happen next? – hslc Jul 26 '10 at 21:34
I'm not a statistician, but from my understanding I'd say, yes, that part of what statistics does. – Mark Meckes Jul 27 '10 at 0:10
great answer! +1 for Persi and Mark – robin girard Jul 27 '10 at 7:21
Induction vs Deduction? – Paolo Jul 27 '10 at 9:14
Like Paolo said, probability theory is mainly concerned with the deductive part, statistics with the inductive part of modeling processes with uncertainty. Perhaps it's interesting to mention that if one thinks that the plausible inductive reasoning should be consistent, then actually the result is bayesian statistics, and more interesting this can be derived from probability theory. So bayesian statistics is basically applied probability theory so to speak. – Thies Heidecke May 27 '11 at 2:08

I like the example of a jar of red and green jelly beans.

A probabilist starts by knowing the proportion of each and asks the probability of drawing a red jelly bean. A statistician infers the proportion of red jelly beans by sampling from the jar.

-
+1 I really like this explanation and think it is very clear. – Tony Breyal Jul 27 '10 at 12:16

It's misleading to simply say that statistics is simply the inverse of probability. Yes, statistical questions are questions of inverse probability, but they are ill-posed inverse problems, and this makes a big difference in terms of how they are addressed.

Probability is a branch of pure mathematics--probability questions can be posed and solved using axiomatic reasoning, and therefore there is one correct answer to any probability question.

Statistical questions can be converted to probability questions by the use of probability models. Once we make certain assumptions about the mechanism generating the data, we can answer statistical questions using probability theory. HOWEVER, the proper formulation and checking of these probability models is just as important, or even more important, than the subsequent analysis of the problem using these models.

One could say that statistics comprises of two parts. The first part is the question of how to formulate and evaluate probabilistic models for the problem; this endeavor lies within the domain of "philosophy of science". The second part is the question of obtaining answers after a certain model has been assumed. This part of statistics is indeed a matter of applied probability theory, and in practice, contains a fair deal of numerical analysis as well.

-

Table 3.1 of Intuitive Biostatistics answers this question with the diagram shown below. Note that all the arrows point to the right for probability, and point to the left for statistics.

PROBABILITY

General ---> Specific

Population ---> Sample

Model ---> Data

STATISTICS

General <--- Specific

Population <--- Sample

Model <--- Data

-
So statistics is synonymous with data analysis? – hslc Jul 26 '10 at 21:25
I don't see any distinction. – Harvey Motulsky Jul 26 '10 at 21:39
Some data analysis does not rely on frequentist statistics. – Fr. Mar 21 '11 at 22:41

Probability is a pure science (math), statistics is about data. They are connected since probability forms some kind of fundament for statistics, providing basic ideas.

-
So probability is pure mathematics and statistics is applied mathematics? – hslc Jul 26 '10 at 21:24
Statistics may be applied and may be not; still the concept of data is always present. – mbq Jul 26 '10 at 21:42

I like this from Steve Skienna's Calculated Bets (see the link for complete discussion):

In summary, probability theory enables us to find the consequences of a given ideal world, while statistical theory enables us to to measure the extent to which our world is ideal.

-

-
Neat and precise! – bludger Sep 5 '13 at 9:33
By this definition, though, a prediction interval is probability rather than statistics. – Glen_b Aug 17 '15 at 10:13

Probability is about quantifying uncertainty whereas statistics is explaining the variation in some measure of interest (e.g., why do income levels vary?) that we observe in the real world.

We explain the variation by using some observable factors (e.g., gender, education level, age etc for the income example). However, since we cannot possibly take into account all possible factors that affect income, we leave any unexplained variation to random errors (which is where quantifying uncertainty comes in).

Since, we attribute "Variation = Effect of Observable Factors + Effect of Random Errors" we need the tools provided by probability to account for the effect of random errors on the variation that we observe.

Some examples follow:

Quantifying Uncertainty

Example 1: You roll a 6-sided dice. What is the probability of obtaining a 1?

Example 2: What is the probability that the annual income of an adult person selected at random from the United States is less than $40,000? Explaining Variation Example 1: We observe that the annual income of a person varies. What factors explain the variation in a person's income? Clearly, we cannot account for all factors. Thus, we attribute a person's income to some observable factors (e.g, education level, gender, age etc) and leave any remaining variation to uncertainty (or in the language of statistics: to random errors). Example 2: We observe that some consumers choose Tide most of the time they buy a detergent whereas some other consumers choose detergent brand xyz. What explains the variation in choice? We attribute the variation in choices to some observable factors such as price, brand name etc and leave any unexplained variation to random errors (or uncertainty). - What if the random errors become greater than the observable factors over time? – hslc Jul 27 '10 at 12:59 In that case you re-work your model as it is no longer consistent with reality. – user28 Jul 28 '10 at 1:06 Probability is the embrace of uncertainty, while statistics is an empirical, ravenous pursuit of the truth (damned liars excluded, of course). - Here I am thinking of all of frequentist/bayesian probability and all of descriptive/exploratory/inferential statistics. – user1108 Sep 14 '10 at 21:57 Similar to what Mark said, Statistics was historically called Inverse Probability, since statistics tries to infer the causes of an event given the observations, while probability tends to be the other way around. - The probability of an event is its long-run relative frequency. So it's basically telling you the chance of, for example, getting a 'head' on the next flip of a coin, or getting a '3' on the next roll of a die. A statistic is any numerical measure computed from a sample of the population. For example, the sample mean. We use this as a statistic which estimates the population mean, which is a parameter. So basically it's giving you some kind of summary of a sample. • You can only get a statistic from a sample, otherwise if you compute a numerical measure on a population, it is called a population parameter. - Probability studies, well, how probable events are. You intuitively know what probability is. Statistics is the study of data: showing it (using tools such as charts), summarizing it (using means and standard deviations etc.), reaching conclusions about the world from which that data was drawn (fitting lines to data etc.), and -- this is key -- quantifying how sure we can be about our conclusions. In order to quantify how sure we can be about our conclusions we need to use Probability. Let's say you have last year's data about rainfall in the region where you live and where I live. Last year it rained an average of 1/4 inch per week where you live, and 3/8 inch where I live. So we can say that rainfall in my region is on average 50% greater than where you live, right? Not so fast, Sparky. It could be a coincidence: maybe it just happened to rain a lot last year where I live. We can use Probability to estimate how confident we can be in our conclusion that my home is 50% soggier than yours. So basically you can say that Probability is the mathematical foundation for the Theory of Statistics. - In probability theory, we are given random variables X1, X2, ... in some way, and then we study their properties, i.e. calculate probability P{ X1 \in B1 }, study the convergence of X1, X2, ... etc. In mathematical statistics, we are given n realizations of some random variable X, and set of distributions D; the problem is to find amongst distributions from D one which is most likely to generate the data we observed. - So we can only find patterns that we were looking for in the first place? – hslc Jul 27 '10 at 12:49 In probability, the distribution is known and knowable in advance - you start with a known probability distribution function (or similar), and sample from it. In statistics, the distribution is unknown in advance. It may even be unknowable. Assumptions are hypothesised about the probability distribution behind observed data, in order to be able to apply probability theory to that data in order to know whether a null hypothesis about that data can be rejected or not. There is a philosophical discussion about whether there is such a thing as probability in the real world, or whether it is an ideal figment of our mathematical imaginations, and all our observations can only be statistical. - Statistics is the pursuit of truth in the face of uncertainty. Probability is the tool that allows us to quantify uncertainty. (I have provided another, longer, answer that assumed that what was being asked was something along the lines of "how would you explain it to your grandmother?") - Answer #1: Statistics is parametrized Probability. Any book on measure-theoretic Probability will tell you about the Probability triplet:$(\Omega, \mathcal F, P)$. But if you're doing Statistics, you have to add$\theta$to the above:$(\Omega, \mathcal F, P_\theta)$, i.e. for different values of$\theta$, you get different probability measures (different distributions). Answer #2: Probability is about going forward; Statistics is about going backward. Probability is about the process of generating (simulating) data given a value of$\theta$. Statistics is about the process of taking data to draw conclusions about$\theta$. Disclaimer: the above are mathematical answers. In reality, much of Statistics is also about designing/discovering appropriate models, questioning existing models, designing experiments, dealing with imperfect data, etc. "All models are wrong." - Analogously, if asked "what is chemistry?" we could reply that it's a set of differential equations. A description of the mathematical theory can give us a small idea of what a subject is about, but it is not the subject itself. – whuber Feb 12 '13 at 17:29 Probability: Given known parameters, find the probability of observing a particular set of data. Statistics: Given a particular set of observed data, make an inference about what the parameters might be. Statistics is "more subjective" and "more art than science" (relative to probability).  $$\underline{\text{Example}}$$ We have a coin that can be flipped. Let$p$be the proportion of coin-flips that are heads. Probability: Suppose$p=\frac{1}{2}$. Then what's the probability of getting$HHH$(three heads in a row)? Most probabilists would give the same, simple answer: "The probability is$\frac{1}{8}$." Statistics: Suppose we get$HHH$. Then what's$p\$?

Different statisticians will give different, often long-winded answers.

-