What's the difference between probability and statistics?

Question

What's the difference between probability and statistics, and why are they studied together?

Answer 1

The short answer to this I've heard from Persi Diaconis is the following:

The problems considered by probability and statistics are inverse to each other. In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens. In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations.

Answer 2

I like the example of a jar of red and green jelly beans.

A probabilist starts by knowing the proportion of each and asks the probability of drawing a red jelly bean. A statistician infers the proportion of red jelly beans by sampling from the jar.

Answer 3

It's misleading to simply say that statistics is simply the inverse of probability. Yes, statistical questions are questions of inverse probability, but they are ill-posed inverse problems, and this makes a big difference in terms of how they are addressed.

Probability is a branch of pure mathematics--probability questions can be posed and solved using axiomatic reasoning, and therefore there is one correct answer to any probability question.

Statistical questions can be converted to probability questions by the use of probability models. Once we make certain assumptions about the mechanism generating the data, we can answer statistical questions using probability theory. HOWEVER, the proper formulation and checking of these probability models is just as important, or even more important, than the subsequent analysis of the problem using these models.

One could say that statistics comprises of two parts. The first part is the question of how to formulate and evaluate probabilistic models for the problem; this endeavor lies within the domain of "philosophy of science". The second part is the question of obtaining answers after a certain model has been assumed. This part of statistics is indeed a matter of applied probability theory, and in practice, contains a fair deal of numerical analysis as well.

See: http://bactra.org/reviews/error/

Answer 4

I like this from Steve Skienna's Calculated Bets (see the link for complete discussion):

In summary, probability theory enables us to find the consequences of a given ideal world, while statistical theory enables us to to measure the extent to which our world is ideal.

Answer 5

Table 3.1 of Intuitive Biostatistics answers this question with the diagram shown below. Note that all the arrows point to the right for probability, and point to the left for statistics.

PROBABILITY

General ---> Specific

Population ---> Sample

Model ---> Data

STATISTICS

General <--- Specific

Population <--- Sample

Model <--- Data

Answer 6

Probability is a pure science (math), statistics is about data. They are connected since probability forms some kind of fundament for statistics, providing basic ideas.

Answer 7

Probability answers questions about what will happen, statistics answers questions about what did happen.

Answer 8

Probability is about quantifying uncertainty whereas statistics is explaining the variation in some measure of interest (e.g., why do income levels vary?) that we observe in the real world.

We explain the variation by using some observable factors (e.g., gender, education level, age etc for the income example). However, since we cannot possibly take into account all possible factors that affect income, we leave any unexplained variation to random errors (which is where quantifying uncertainty comes in).

Since, we attribute "Variation = Effect of Observable Factors + Effect of Random Errors" we need the tools provided by probability to account for the effect of random errors on the variation that we observe.

Some examples follow:

Quantifying Uncertainty

Example 1: You roll a 6-sided die. What is the probability of obtaining a 1?

Example 2: What is the probability that the annual income of an adult person selected at random from the United States is less than $40,000?

Explaining Variation

Example 1: We observe that the annual income of a person varies. What factors explain the variation in a person's income?

Clearly, we cannot account for all factors. Thus, we attribute a person's income to some observable factors (e.g, education level, gender, age etc) and leave any remaining variation to uncertainty (or in the language of statistics: to random errors).

Example 2: We observe that some consumers choose Tide most of the time they buy a detergent whereas some other consumers choose detergent brand xyz. What explains the variation in choice? We attribute the variation in choices to some observable factors such as price, brand name etc and leave any unexplained variation to random errors (or uncertainty).

Answer 9

Similar to what Mark said, Statistics was historically called Inverse Probability, since statistics tries to infer the causes of an event given the observations, while probability tends to be the other way around.

Answer 10

Probability is the embrace of uncertainty, while statistics is an empirical, ravenous pursuit of the truth (damned liars excluded, of course).

Answer 11

The probability of an event is its long-run relative frequency. So it's basically telling you the chance of, for example, getting a 'head' on the next flip of a coin, or getting a '3' on the next roll of a die.

A statistic is any numerical measure computed from a sample of the population. For example, the sample mean. We use this as a statistic which estimates the population mean, which is a parameter. So basically it's giving you some kind of summary of a sample.

• You can only get a statistic from a sample, otherwise if you compute a numerical measure on a population, it is called a population parameter.

Answer 12

Probability studies, well, how probable events are. You intuitively know what probability is.

Statistics is the study of data: showing it (using tools such as charts), summarizing it (using means and standard deviations etc.), reaching conclusions about the world from which that data was drawn (fitting lines to data etc.), and -- this is key -- quantifying how sure we can be about our conclusions.

In order to quantify how sure we can be about our conclusions we need to use Probability. Let's say you have last year's data about rainfall in the region where you live and where I live. Last year it rained an average of 1/4 inch per week where you live, and 3/8 inch where I live. So we can say that rainfall in my region is on average 50% greater than where you live, right? Not so fast, Sparky. It could be a coincidence: maybe it just happened to rain a lot last year where I live. We can use Probability to estimate how confident we can be in our conclusion that my home is 50% soggier than yours.

So basically you can say that Probability is the mathematical foundation for the Theory of Statistics.

Answer 13

In probability theory, we are given random variables X1, X2, ... in some way, and then we study their properties, i.e. calculate probability P{ X1 \in B1 }, study the convergence of X1, X2, ... etc.

In mathematical statistics, we are given n realizations of some random variable X, and set of distributions D; the problem is to find amongst distributions from D one which is most likely to generate the data we observed.

Answer 14

In probability, the distribution is known and knowable in advance - you start with a known probability distribution function (or similar), and sample from it.

In statistics, the distribution is unknown in advance. It may even be unknowable. Assumptions are hypothesised about the probability distribution behind observed data, in order to be able to apply probability theory to that data in order to know whether a null hypothesis about that data can be rejected or not.

There is a philosophical discussion about whether there is such a thing as probability in the real world, or whether it is an ideal figment of our mathematical imaginations, and all our observations can only be statistical.

Answer 15

Statistics is the pursuit of truth in the face of uncertainty. Probability is the tool that allows us to quantify uncertainty.

(I have provided another, longer, answer that assumed that what was being asked was something along the lines of "how would you explain it to your grandmother?")

Answer 16

Probability: Given known parameters, find the probability of observing a particular set of data.

Statistics: Given a particular set of observed data, make an inference about what the parameters might be.

Statistics is "more subjective" and "more art than science" (relative to probability).

$$$$

$$\underline{\text{Example}}$$

We have a coin that can be flipped. Let $p$ be the proportion of coin-flips that are heads.

Probability: Suppose $p=\frac{1}{2}$. Then what's the probability of getting $HHH$ (three heads in a row)?

Most probabilists would give the same, simple answer: "The probability is $\frac{1}{8}$."

Statistics: Suppose we get $HHH$. Then what's $p$?

Different statisticians will give different, often long-winded answers.

Answer 17

Answer #1: Statistics is parametrized Probability. Any book on measure-theoretic Probability will tell you about the Probability triplet: $(\Omega, \mathcal F, P)$. But if you're doing Statistics, you have to add $\theta$ to the above: $(\Omega, \mathcal F, P_\theta)$, i.e. for different values of $\theta$, you get different probability measures (different distributions).

Answer #2: Probability is about going forward; Statistics is about going backward. Probability is about the process of generating (simulating) data given a value of $\theta$. Statistics is about the process of taking data to draw conclusions about $\theta$.

Disclaimer: the above are mathematical answers. In reality, much of Statistics is also about designing/discovering appropriate models, questioning existing models, designing experiments, dealing with imperfect data, etc. "All models are wrong."

Answer 18

The difference between probabilities and statistics is that in probabilities there is no mistake. We are sure for the probability because we know exactly how many sides has a coin, or how many blue caramels are in the vase. But in statistics we examine a piece of a population of whatever we examine, and from this, we try to see the truth, but always there is an a% of wrong conclusions. The only thing in statistics that is true, is this a% mistake, that in fact is a probability.

Answer 19

Savage's text Foundations of Statistics has been cited over 12000 times on Google Scholar.[3] It tells the following.

It is unanimously agreed that statistics depends somehow on probability. But, as to what probability is and how it is connected with statistics, there has seldom been such complete disagreement and breakdown of communication since the Tower of Babel. Doubtless, much of the disagreement is merely terminological and would disappear under sufficiently sharp analysis.

https://en.wikipedia.org/wiki/Foundations_of_statistics

So the point that Probability Theory is a Foundation of Statistics is hardly disputed. Everything else is fair game.

But in trying to be more helpful, practical with an answer...

However, probability theory contains much that is mostly of mathematical interest and not directly relevant to statistics. Moreover, many topics in statistics are independent of probability theory

https://en.wikipedia.org/wiki/Probability_and_statistics

The above is not exhaustive or authorative by any means, but I believe it's useful.

Commonly it has helped me to see things such as...

Descrete Mathematics >> Probability Theory >> Statistics

With each being heavily used, on average, in the foundations of the next. That is there are large intersections in how we study the next's foundations.

PS. There's inductive and deductive Statistics, so that's not where the difference lies.

Answer 20

The term "statistics" is beautifully explained by J. C. Maxwell in the article Molecules (in Nature 8, 1873, pp. 437–441). Let me quote the relevant passage:

When the working members of Section F get hold of a Report of the Census, or any other document containing the numerical data of Economic and Social Science, they begin by distributing the whole population into groups, according to age, income-tax, education, religious belief, or criminal convictions. The number of individuals is far too great to allow of their tracing the history of each separately, so that, in order to reduce their labour within human limits, they concentrate their attention on small number of artificial groups. The varying number of individuals in each group, and not the varying state of each individual, is the primary datum from which they work.

This, of course, is not the only method of studying human nature. We may observe the conduct of individual men and compare it with that conduct which their previous character and their present circumstances, according to the best existing theory, would lead us to expect. Those who practise this method endeavour to improve their knowledge of the elements of human nature, in much the same way as an astronomer corrects the elements of a planet by comparing its actual position with that deduced from the received elements. The study of human nature by parents and schoolmasters, by historians and statesmen, is therefore to be distinguished from that carried on by registrars and tabulators, and by those statesmen who put their faith in figures. The one may be called the historical, and the other the statistical method.

The equations of dynamics completely express the laws of the historical method as applied to matter, but the application of these equations implies a perfect knowledge of all the data. But the smallest portion of matter which we can subject to experiment consists of millions of molecules, not one of which ever becomes individually sensible to us. We cannot, therefore, ascertain the actual motion of any one of these molecules, so that we are obliged to abandon the strict historical method, and to adopt the statistical method of dealing with large groups of molecules.

He gives this explanation of the statistical method in several other works. For example, "In the statistical method of investigation, we do not follow the system during its motion, but we fix our attention on a particular phase, and ascertain whether the system is in that phase or not, and also when it enters the phase and when it leaves it" (Trans. Cambridge Philos. Soc. 12, 1879, pp. 547–570).

There's another beautiful passage by Maxwell about "probability" (from a letter to Campbell, 1850, reprinted in The Life of James Clerk Maxwell, p. 143):

the actual science of Logic is conversant at present only with things either certain, impossible, or entirely doubtful, none of which (fortunately) we have to reason on. Therefore the true Logic for this world is the Calculus of Probabilities, which takes account of the magnitude of the probability (which is, or which ought to be in a reasonable man's mind).

So we can say:

– In statistics we are "concentrating our attention on small number of artificial groups" or quantities; we're making a sort of cataloguing or census.

– In probability we are calculating our uncertainty about some events or quantities.

The two are distinct, and we can be doing the one without the other.

For example, if we make a complete census of the entire population of a nation and count the exact number of people belonging to particular groups such as age, gender, and so on, we are doing statistics. There's no uncertainty – probability – involved, because the numbers we find are exact and known.

On the other hand, imagine someone passing in front of us on the street, and we wonder about their age. In this case we're uncertain and we use probability, but there is no statistics involved, since we aren't making some sort of census or catalogue.

But the two can also occur together. If we can't make a complete census of a population, we have to guess how many people are in specific age-gender groups. Hence we're using probability while doing statistics. Vice versa, we can consider exact statistical data about people's ages, and from such data try to make a better guess about the person passing in front of us. Hence we're using statistics while deciding upon a probability.

Answer 21

Many people and mathematicians say that 'STATISTICS is the inverse of PROBABILITY',but its not particularly right. The way of approaching or method of solving these 2 are completely different but they are INTERCONNECTED.

i will like to refer to my friend John D Cook.....

"I like the example of a jar of red and green jelly beans.

A probabilist starts by knowing the proportion of each and lets say finds the probability of drawing a red jelly bean. A statistician infers the proportion of red jelly beans by sampling from the jar."

Now the proportion of the red jelly bean obtained by sampling from the jar is used by the probabilist to find the probability of drawing a red bean from the jar

Consider this example---->>>

In an examination 30% of students failed in physics, 25% failed in maths, 12% failed both in physics and maths. A student is selected at random find the probability that the student has failed in Physics,if it is known that he failed in maths.

The above sum is a problem of probability,but if we look carefully we will find that the sum is provided with some statistical data

30% student failed in physics, 25% " " " maths ' ' ' These are basically frequencies if the percentages are calculated. thus we are being provided with a statistical data which in turn helps us to find the probability

SO PROBABILITY AND STATISTICS ARE VERY MUCH INTERCONNECTED OR RATHER WE CAN SAY THAT PROBABILITY IS DEPENDENT A LOT ON STATISTICS