What's the difference between probability and statistics, and why are they studied together?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
|
||||
|
|
|
The short answer to this I've heard from Persi Diaconis is the following: the problems considered by probability and statistics are inverse to each other. In probability theory we consider some underlying process which has some randomness or uncertainty modeled by random variables, and we figure out what happens. In statistics we observe something that has happened, and try to figure out what underlying process would explain those observations. |
|||||||||||||||
|
|
I like the example of a jar of red and green jelly beans. A probabilist starts by knowing the proportion of each and asks the probability of drawing a red jelly bean. A statistician infers the proportion of red jelly beans by sampling from the jar. |
|||||
|
|
Table 3.1 of Intuitive Biostatistics answers this question with the diagram shown below. Note that all the arrows point to the right for probability, and point to the left for statistics. PROBABILITY
STATISTICS
|
|||||||
|
|
|
I like this from Steve Skienna's Calculated Bets (see the link for complete discussion):
|
|||
|
|
|
Probability is a pure science (math), statistics is about data. They are connected since probability forms some kind of fundament for statistics, providing basic ideas. |
|||
|
Similar to what Mark said, Statistics was historically called Inverse Probability, since statistics tries to infer the causes of an event given the observations, while probability tends to be the other way around. |
|||
|
|
|
Probability is the embrace of uncertainty, while statistics is an empirical, ravenous pursuit of the truth (damned liars excluded, of course). |
|||
|
|
It's misleading to simply say that statistics is simply the inverse of probability. Yes, statistical questions are questions of inverse probability, but they are ill-posed inverse problems, and this makes a big difference in terms of how they are addressed. Probability is a branch of pure mathematics--probability questions can be posed and solved using axiomatic reasoning, and therefore there is one correct answer to any probability question. Statistical questions can be converted to probability questions by the use of probability models. Once we make certain assumptions about the mechanism generating the data, we can answer statistical questions using probability theory. HOWEVER, the proper formulation and checking of these probability models is just as important, or even more important, than the subsequent analysis of the problem using these models. One could say that statistics comprises of two part. The first part is the question of how to formulate and evaluate probabilistic models for the problem, and this endeavor lies within the domain of "philosophy of science." The second part is the question of obtaining answers after a certain model has been assumed. This part of statistics is indeed a matter of applied probability theory, and in practice, a fair deal of numerical analysis as well. |
|||
|
|
|
Probability answers questions about what will happen, statistics answers questions about what did happen. |
|||
|
|
|
Probability is about quantifying uncertainty whereas statistics is explaining the variation in some measure of interest (e.g., why do income levels vary?) that we observe in the real world. We explain the variation by using some observable factors (e.g., gender, education level, age etc for the income example). However, since we cannot possibly take into account all possible factors that affect income, we leave any unexplained variation to random errors (which is where quantifying uncertainty comes in). Since, we attribute "Variation = Effect of Observable Factors + Effect of Random Errors" we need the tools provided by probability to account for the effect of random errors on the variation that we observe. Some examples follow: Quantifying Uncertainty Example 1: You roll a 6-sided dice. What is the probability of obtaining a 1? Example 2: What is the probability that the annual income of an adult person selected at random from the United States is less than $40,000? Explaining Variation Example 1: We observe that the annual income of a person varies. What factors explain the variation in a person's income? Clearly, we cannot account for all factors. Thus, we attribute a person's income to some observable factors (e.g, education level, gender, age etc) and leave any remaining variation to uncertainty (or in the language of statistics: to random errors). Example 2: We observe that some consumers choose Tide most of the time they buy a detergent whereas some other consumers choose detergent brand xyz. What explains the variation in choice? We attribute the variation in choices to some observable factors such as price, brand name etc and leave any unexplained variation to random errors (or uncertainty). |
|||||
|
|
The probability of an event is its long-run relative frequency. So it's basically telling you the chance of, for example, getting a 'head' on the next flip of a coin, or getting a '3' on the next roll of a die. A statistic is any numerical measure computed from a sample of the population. For example, the sample mean. We use this as a statistic which estimates the population mean, which is a parameter. So basically it's giving you some kind of summary of a sample.
|
||||
|
|
|
Probability studies, well, how probable events are. You intuitively know what probability is. Statistics is the study of data: showing it (using tools such as charts), summarizing it (using means and standard deviations etc.), reaching conclusions about the world from which that data was drawn (fitting lines to data etc.), and -- this is key -- quantifying how sure we can be about our conclusions. In order to quantify how sure we can be about our conclusions we need to use Probability. Let's say you have last year's data about rainfall in the region where you live and where I live. Last year it rained an average of 1/4 inch per week where you live, and 3/8 inch where I live. So we can say that rainfall in my region is on average 50% greater than where you live, right? Not so fast, Sparky. It could be a coincidence: maybe it just happened to rain a lot last year where I live. We can use Probability to estimate how confident we can be in our conclusion that my home is 50% soggier than yours. So basically you can say that Probability is the mathematical foundation for the Theory of Statistics. |
|||
|
|
|
In probability theory, we are given random variables X1, X2, ... in some way, and then we study their properties, i.e. calculate probability P{ X1 \in B1 }, study the convergence of X1, X2, ... etc. In mathematical statistics, we are given n realizations of some random variable X, and set of distributions D; the problem is to find amongst distributions from D one which is most likely to generate the data we observed. |
|||
|
|
Statistics is the pursuit of truth in the face of uncertainty. Probability is the tool that allows us to quantify uncertainty. (I have provided another, longer, answer that assumed that what was being asked was something along the lines of "how would you explain it to your grandmother?") |
|||
|
|
|
In probability, the distribution is known and knowable in advance - you start with a known probability distribution function (or similar), and sample from it. In statistics, the distribution is unknown in advance. It may even be unknowable. Assumptions are hypothesised about the probability distribution behind observed data, in order to be able to apply probability theory to that data in order to know whether a null hypothesis about that data can be rejected or not. There is a philosophical discussion about whether there is such a thing as probability in the real world, or whether it is an ideal figment of our mathematical imaginations, and all our observations can only be statistical. |
|||
|
|
|
Answer #1: Statistics is parametrized Probability. Any book on measure-theoretic Probability will tell you about the Probability triplet: $(\Omega, \mathcal F, P)$. But if you're doing Statistics, you have to add $\theta$ to the above: $(\Omega, \mathcal F, P_\theta)$, i.e. for different values of $\theta$, you get different probability measures (different distributions). Answer #2: Probability is about going forward; Statistics is about going backward. Probability is about the process of generating (simulating) data given a value of $\theta$. Statistics is about the process of taking data to draw conclusions about $\theta$. Disclaimer: the above are mathematical answers. In reality, much of Statistics is also about designing/discovering appropriate models, questioning existing models, designing experiments, dealing with imperfect data, etc. "All models are wrong." |
|||||
|
