# Why is statistics useful when many things that matter are one shot things?

I don't know if it's just me, but I am very skeptical of statistics in general. I can understand it in dice games, poker games, etc. Very small, simple, mostly self-contained repeated games are fine. For example, a coin landing on its edge is small enough to accept the probability that landing heads or tails is ~50%.

Playing a $10 game of poker aiming for a 95% win is fine. But what if your entire life savings + more is dependent on you hitting a win or not? How would knowing that you'd win in 95% of the time in that situation will help me at all? Expected value doesn't help much there. Other examples include a life-threatening surgery. How does that help knowing that it is 51% survival rate versus 99% survival rate given existing data? In both cases, I don't think it will matter to me what the doctor tells me, and I would go for it. If actual data is 75%, he might as well tell me (barring ethics and law), that there is a 99.99999% chance of survival so I'd feel better. In other words, existing data doesn't matter except binomially. Even then, it doesn't matter if there is a 99.99999% survival rate, if I end up dying from it. Also, earthquake probability. It doesn't matter if a strong earthquake happened every x (where x > 100) years on average. I have no idea if an earthquake will happen ever in my lifetime. So why is it even useful information? A less serious example, say, 100% of the places I've been to that I love are in the Americas, indifferent to 100% of the places I've been to in Europe, and hate 100% of the places that I have been to in Asia. Now, that by no means mean that I wouldn't find a place that I love in Asia on my next trip or hate in Europe or indifferent in America, just by the very nature that the statistics doesn't capture all of the information I need, and I probably can never capture all of the information I need, even if I have traveled to over x% of all of those continents. Just because there are unknowns in the 1-x% of those continents that I haven't been to. (Feel free to replace the 100% with any other percentage). I understand that there is no way to brute force everything and that you have to rely on statistics in many situations, but how can we believe that statistics are helpful in our one shot situation, especially when statistics basically do not extrapolate to outlier events? Any insights to get over my skepticism of statistics? • (+1) Welcome to our site! It isn't just you: this is a deep question that goes to the foundations of statistics. – whuber Sep 7 '12 at 22:06 • The "life savings" example mixes separate issues. In economics, a common model for rational risk aversion is to maximize expected utility, not expected money, where utility is typically a concave (sublinear) function like log(money). This means losses cost more than gains of the same size, and this effect is larger for larger changes. This is very different from not believing there is any difference between$50\%$and$99\%\$, which leads to inconsistent and irrational behavior. – Douglas Zare Sep 8 '12 at 7:28
• @DouglasZare this sounds like a very interesting area. Can you provide an introductory article to the topic of individual risk aversion regarding live savings ? – steffen Sep 8 '12 at 8:37
• @steffen: This material is covered in many basic economics texts. The theory of expected utility maximization is viewed by many as too simple, and insufficient to explain many phenomena, but it is an important starting point to understand before moving on to ideas such as prospect theory. Something which is easily explained by expected utility maximization instead of expected money maximization should not be viewed as a failure of probability theory. en.wikipedia.org/wiki/Expected_utility_hypothesis – Douglas Zare Sep 8 '12 at 8:46
• This strikes me as an important topic & discussion that could stay open. (I would understand if people wanted to make it CW, though.) – gung - Reinstate Monica Dec 15 '15 at 1:49

First I think that you may be confusing "statistics" meaning a collection of numbers or other facts describing a group or situation, and "statistics" meaning the science of using data and information to understand the world in the face of variation (others may be able to improve on my definitions). Statisticians use both senses of the word, so it is not surprising when people mix them up.

Statistics (the science) is a lot about choosing strategies and choosing the best strategy even if we only get to apply it once. Some times when I (and others) teach probability we use the classic Monty Hall problem (3 doors, 2 goats, 1 car) to motivate it and we show how we can estimate probabilities by playing the game a bunch of times (not for prizes) and we can see that the "switch" strategy wins 2/3 of the time and the "stay" strategy only wins 1/3 of the time. Now if we had the opportunity to play the game a single time we would know some things about which strategy gives a better chance of winning.

The surgery example is similar, you will only have the surgery (or not have the surgery) once, but don't you want to know which strategy benifits more people? If your choices are surgery with some chance greater than 0% of survival or no surgery and 0% of survival, then yes there is little difference between the surgery having 51% survival and 99.9% survival. But what if there are other options as well, you can choose between surgery, doing nothing (which has 25% survival) or a change of diet and exercise which has 75% survival (but requires effort on your part), no wouldn't you care about if the surgery option has 51% vs. 99% survival?

Also consider the doctor, he will be doing more than just your surgery. If surgery has 99.9% survival then he has no reason to consider alternatives, but if it only has 51% survival then while it may be the best choice today, he should be looking for other alternatives that increase that survival. Yes even with 90% survival he will loose some patients, but which strategy gives him the best chance of saving the most patients?

This morning I wore my seat belt while driving (my usual strategy), but did not get in any accidents, so was my strategy a waste of time? If I knew when I would get in an accident then I could save time by only putting on the seat belt on those occasions and not on others. But I don't know when I will be in an accident so I will stick with my wear the seat belt strategy because I believe it will give me the best chance if I ever am in an accident even if that means wasting a bit of time and effort in the high percentage (hopefully 100%) of times that there is no accident.

• +1 Greg Good post! I was writing mine at the same time as you. We may overlap a little but I think we both had things to say that were right on the mark and did not overlap. I am not sure what the OP thinks statistics is. It is nice that you gave him the benefit of the doubt. I took a more angry approach to it. – Michael R. Chernick Sep 7 '12 at 21:55
• Hi Greg, I liked your answer, but can I reason it like this: statistics (the science) is itself a statistic, it works for x% of the time, (possibly high x), but there are the 1-x% unknown/random factors that we always need to be aware of. Given that we can model the unknown in any # (possibly infinite) of ways, we will never know x. Hopefully these outliers will never happen, but we should always be mindful and err on the conservative, especially if the event is catastrophic (ie. asteroids, financial products, nuclear accidents for society and car accidents for personal). Does this make sense? – statskeptic Sep 10 '12 at 18:07
• @statskeptic, what you say applies to all areas, not just statistics. In fact is applies less to true statistics than other fields because when statistics is done right the assumptions are clear. Most times that statistics has failed it has not been the techniques, but that they were applied incorrectly. In any field that involves uncertainty (which is pretty much anything other than religion or pure math, and even they have some) you can have an answer that is either wrong, useless, or uses statistics. – Greg Snow Sep 11 '12 at 16:55
• (+1) for statistics being "the science of using data and information to understand the world in the face of variation" – half-pass Sep 6 '14 at 19:52

Just because you don't use statistics in your daily life does not mean that the field does not directly affect you. When you are at the doctor and they recommend one treatment over the other, you can bet that behind that recommendation was many clinical trials that used statistics to interpret the results of their experiments.

It turns out that the concept of expected value is also very useful even if you do not personally use the concept. Your example of betting your life savings fails to take into account how risk adverse you are. Other situations might find yourself less risk adverse, or where there are not catastrophic outcomes. Business, finance, actuarial contexts and others are examples of this. Perhaps you are issuing home insurance policy - then all of the sudden knowing the probability of an earthquake occurring within some specified period of time matters a great deal.

In the end statistics is a great way to deal with uncertainty. Your last example you made up some data about places you like to travel and claimed that statistics will say that you will never find a place in Asia that you like. This is just wrong. Of course this data will make you believe that Asia is less likely to have a place you like, but you can set your prior belief to be whatever you like, and statistics will tell you how to update your belief given the new data. Furthermore, it allows you to do modify your belief in a principled way that will allow you to act rationally in the presence of uncertainty.

• The traveling example was just a made up one, but the idea is that statistics does not capture the unknown. Your example of business contexts made me thought of the example of insurance companies of WTC probably estimated the cost/benefit of insuring the building without taking into account of planes destroying the building, but yet it is the one that mattered most. – statskeptic Sep 7 '12 at 22:11
• +1 @jjund3 for addressing the OPs specific questions and for intermingling Bayesian and frequentist staistics without any conflict. – Michael R. Chernick Sep 7 '12 at 22:55
• @statskeptic Your point that statistics cannot account for all possible uncertainties is a good one. But it doesn't have to be complete and perfect to be useful. We do have knowledge about terrorists. Prior to 9/11 we did have examples of terrorists going on suicide missions and we had experience with highjacking of planes. The information could have been pieced together to determine that crashing aplane into the World Trade Center was a possibility although we would probably have assessed it as a remote possibility. – Michael R. Chernick Sep 7 '12 at 23:02
• We knew the World Trade Center was a favorite terrist target. It had been attacked once before with a bomb set off in the basement. The fact that the bomb was not strong enough to do the desired damage was at least a hint that the next time some very different method would be used. Of course as often said hind sight is 20-20. There are many examples where the unexpected or improbable happens. But not in the case of the Challanger disaster. There the Thiokol engineers even with limited data knew that there was some riks of a catastrophic failure due to O-rings failure at low temperature. – Michael R. Chernick Sep 7 '12 at 23:07
• @statskeptic Your argument is very similar to Taleb's skepticism/ bashing of statistics in his book the Black Swan. I think many statisticians myself included have shot holes in his argument which basially says that statistics is useless because it cannot predict that rare and unthinkable event (9/11 in your example,the stock market crash in his). – Michael R. Chernick Sep 7 '12 at 23:19

The world is stochastic not deterministic. If it were deterministic the physicists would be ruling the world and statisticians would be out of a job. But the reality is that statisticians are in high demand in almost every discipline. That is not to say that there isn't a place for physics and other sciences but statistics works hand in hand with science and is the basis for many scientific discoveries.

Enough chatter and down to specifics. I have worked the last 17 years in the medical industry, first in medical devices, then pharmaceuticals, and now general medical research. Drugs and medical devices that improve quality of life and often save or extend life are developed and approved in this country and around the world on a regular basis. In the US approval requires evidence of safety and efficacy before the FDA will allow a drug or medical device to be marketed. Evidence to the FDA comes from clinical trials in phases. All the clinical trials require valid statistical design and analysis methods. Nothing is perfect. Drugs work well for some people while others may not respond or will have adverse events (bad reactions that can cause illness or death). The trials separate out the ineffective drugs from the effective. Most drugs fail and there is often a ten year cycle from early stage development to end of phase III with approval and marketing at the end of the trial. Postmarket surveillance which also requires statistics is then applied to make sure that the drug works well enough for the general population. Sometimes the general population that the drug is approved for is a less restrictive group than the patients that were eligible for the clinical trials. So sometimes drugs do turn out to be dangerous and get pulled from the market. Statistics helps in all aspects of drug safety.

Statistics is not perfect. We live with some mistakes due to randomness and uncertainty. But it is controlled and our lives are better and errors are reduced from what they would be had statistical science not been involved.

• Don't get me wrong. I understand there is statistics in everything, even physics with quantum mechanics is all about probability, and there isn't enough atoms to do computation without statistics. I just want to find out how to deal with the randomness and uncertainty which can influence my (or other people's) life more than any actual statistic or distribution. – statskeptic Sep 7 '12 at 22:00
• Okay statskeptic so you are not confused. But why is it so hard to see how statistics improves your chance for success. Probability theory tells you the odds of winning games of chance. If you could use Thorpe's Beat the Dealer strategy in blackjack and you have a large bank of funds you can make a fortune in the long run. The MIT students proved it in Las Vegas even though the advantage in counting has been reduced by the use of mixing multiple decks. It is true. The casino knows that card counters are a threat. – Michael R. Chernick Sep 7 '12 at 22:16
• They search for them and when they think they find one they throw him or her out of the casino no questions asked. – Michael R. Chernick Sep 7 '12 at 22:17
• Also, please don't think that I'm trying to flame your profession. There are computers that are doing computation statistically to save power, and I respect that. I'm just trying to learn how people with much more knowledge than me in statistics deal with these questions. – statskeptic Sep 7 '12 at 22:19
• @statskeptic I fyou saw my original post I apologize for my inital comments. They were rightfully edited out by a moderator. I think I misunderstood what you were trying to say. I hope we answered your question well and relieved some of your skepticism. – Michael R. Chernick Sep 7 '12 at 22:44

I myself have the same doubts about the usefulness of probability, and statistics, when it comes to taking decision about a single event. In my opinion, knowing the probability, real or estimated, is extremely important when the objective is estimating outcomes of samples, be they a single event repeated a number of times or a sample drown from a certain population. In short, knowing the probability makes more sense for the casino who, based on probability calculations can put the rules that guarantee he would win in the long run (after many plays) and not for a gambler who pretends to play one time, so he would won or loose (these are the outcomes when the experiment is run a single time). It’s also important for the generals who contemplate sending their solders to a battle with the risk (probability) of losing 10% of them, but not for a certain solder (say, John) who is only going to die or survive. There are so many examples like these in real life.

The point I want to make is that, Probability and Statistics, not only are useful in real life but, more precisely, they are a tool for all modern scientific researches and decision making rules. However, it’s not correct to say that rationality implies relying on the probability of a single event, without the intention or the possibility of repeating it, for estimating the outcome. The tendency of the probability to influence a certain individual’s decision, based on her or his degree of risk aversion, is obviously subjective. Risk avert and risk lover have different attitudes (decisions) toward the same lottery (the same expected value).

• The point about risk aversion is an interesting one, in terms of how people react to an uncertain event. But note that when economists consider choice under uncertainty (e.g. commodities contingent upon the state of the world), the true probability comes into play via the fair odds line (a budget constraint reflecting the possible bundles under an actuarially fair gamble). Agents behave not just according to their preferences (e.g. risk aversion) alone, but also according to the interplay of their budget constraint (the available gamble) and their assessment of the fair odds line. – Silverfish Dec 15 '15 at 1:31
• Briefly, it's not true that all risk-averse people "never gamble" (in the broadest sense of the word), just that they can't be tempted to gamble by actuarially fair odds. A sufficient risk premium (dependent on their degree of risk aversion) can alter that decision, however. Since this analysis depends on the agent's perception of the fair odds, even on a "one-off shot" a rational agent will weigh up the probabilities. – Silverfish Dec 15 '15 at 1:35
• 1- I didn’t say that risk averse persons never gamble. 2-What I meant by “subjective” is that knowing the expected value of a lottery does not determine a person’s attitude toward it. All other things equal, this attitude is a function of a personal characteristic which is the degree of risk aversion which determines the expected utility of the gamble. 3-Rationality in economic theory depends upon hypotheses and hence is relative. That’s why two individuals showing different attitudes toward the same expected value could be both called “rational”. – Mohamed Lemine Dec 30 '15 at 13:41
• I wish we don't miss the central point of this discussion, which is dealing with the fact that events with very low probability can take place in any trail. and vise versa. – Mohamed Lemine Dec 30 '15 at 14:10

The long and the short of it is that probability is the unique generalization of ordinary true/false logic to degrees of belief between 0 and 1. This is the so-called logical Bayesian interpretation of probability, originated by R.T. Cox and later championed by E.T. Jaynes.

Furthermore under weak assumptions it can be shown that the right way to order uncertain outcomes by preference is to order them by expected utility, with the expected taken with respect to the probability distribution over outcomes.

See Robert Clemen, "Making Hard Decisions", for an introduction and exposition on applied decision analysis which is based on Bayesian probability and expected utility.

You are absolutely right to be skeptical about conventional frequentist statistics; by the design of its inventors (R.A. Fisher, J. Neyman, E. Pearson) it is limited to repetitive events. But many everyday problems don't involve repetitive events. What to do? The typical approach is some combination of forcing square pegs into round holes, and moving the goalposts. Shameful, really.

• -1 In my opinion a very poor and unfair portrayal of frequentist statistics. I would not take such a negative view of the Bayesian approach. But the Bayesians (any camp) are not free from criticism. Is degree of belief to be the staple of inference? Is degree of belief subjective and personal, so that two people can give two different answers? What about the need for a prior distribution? How should it be picked? Lots of questions for any paradigm for inference. But aren't we past the stage of hostle bickering over the foundations? – Michael R. Chernick Sep 7 '12 at 22:05
• There is more about the scientific method to unite us and say resoundingly that STAISTICS IS IMPORTANT when facing a skeptic. Instead you agree with the skeptic in order to take a cheap shot at frequentist methods! That is what is shameful. – Michael R. Chernick Sep 7 '12 at 22:06
• @MichaelChernick: (1) to simply shout STATISTICS IS IMPORTANT is hardly an argument which will win over a skeptic. (2) Bayesian inference has the same relation to problem data as ordinary logic. That is, given some premises, you crank out a solution by applying laws of probability. The data (e.g. any prior distribution) are neither right nor wrong; they just are. Reasonable people disagree about prior distributions just as they might about any other problem data. – Robert Dodier Sep 7 '12 at 23:24
• I don't mind getting into a discussion about the foundations. This is not the appropriate place and my only point to you was that I think you took a cheap shot and that your answer was inappropriate. That doesn't require a discussion of the foundations of statistics. – Michael R. Chernick Sep 8 '12 at 0:11
• Hug it out guys. – Brandon Bertelsen Sep 8 '12 at 11:54

I skeptical of statistics for the following reasons.

1. I am convinced anybody without a graduate degree in statistics has no clue what they are doing. Unf. there are millions of people across the world doing research without a graduate degree in statistics. I was an undergraduate math major at the Univeristy of Maryland College, Park. I took 4 400 level math classes. All teachers did was teach you how to calculate stuff. Nobody taught me how to make sense of anything or do any statistical analysis except for hypothesis testing, which makes no sense for 2 reasons.
1. For every hypothesis test I was taught, I had to make assumptions beforehand. Nobody taught me which assumption(s) I had to start with. 2. P values make no sense logically. A graduate degree in statistics might teach you what a p value actually is. However, I am convinced no undergraduate knows how to use it. The undergraduate definition assumes a probability of something that depends on the hypothesis being correct. Logically, the definition makes no sense at all. Even worse, NOBODY has ever told me where the probability comes from. I have actually emailed almost my whole math department(more than 200 people) if somebody could give me an answer. The most popular and only responses were "one would have to ASSUME the error rates for the probability" (When I asked people how this was done, they all answered me "from previous experiments", I could not get anything more specific from anybody), "It's just the way it is", and "it is completely random".
The same thing happened when I googled what the signifigance of a p value is. It leads me to the conclusion...

2. Even a sig. number of math and statistics professors have no clue what the logic behind statistics is. I don't expect people to have in depth knowledge. However, I have a feeling that even a sig. % of researches and professors do not understand any of the underlying logic behind statistics.

3. Statistical error is not the same thing as actual error. Because people like to use statistics to derive estimates for things that are humongous, people like to use statistical error to "mask" the fact that they have no clue what the actual error.

4. People use small samples for big populations because statistical theory tells them they can. I learned from one of my college courses, that people like to use data that is an estimate from about 30 schools in the country to show that there are few violent incidents in schools in the whole country. There are about 100,000 schools. That sounds insane. A whole popular movement is based off of about 30 schools in the whole country.

5. People like to make the burden of proof statistical. The Higgs Bossom was never discovered. It was discovered statistically, but that doesn't mean anything. Something being discovered purely statistically is useless because nobody knows the accuracy of statistics.

6. People like to use statistics to make important desicions. Statistics can be used as a guide, but nobody knows how accurate it really is. Just because a problem seems impossible to solve does not mean that statistics is the next best thing. The fact that DNA testing is based off of statistic gives me the chills. Can I be given the death penalty soley because of statistics? Could a murderer be released from jail soley because of statistics?

I believe statistics can be useful, but only if it is not used as the conclusion. I believe statistics can tell us what some of the possibilities are. Then logic, not statistical logic should be used to prove which possibility(s) is correct.

• "...useless because nobody knows the accuracy of statistics", & your complaints about drawing conclusions from statistical evidence like a sample of schools or DNA, suggests you don't trust statistical inference. Yet often a limited sample is all the evidence available, or all the data you can afford to capture. How to weigh up such evidence? We face uncertainty, as our sample won't exactly reflect the wider population. Inference deals with that uncertainty, eg confidence intervals measure uncertainty in sample statistics like the sample mean (roughly, the "accuracy" of statistics is known). – Silverfish May 31 '15 at 8:33
• "professors do not understand any of the underlying logic behind statistics" - there are some radically different philosophies of statistics (see e.g. the Bayesian-Frequentist debate) but most people are pragmatic about the techniques they apply to a particular problem. This may not feature highly, if at all, on an undergrad course, but the philosophy of statistics certainly wasn't drawn up at random on the back of an envelope one day. As for p-values, "Logically, the definition makes no sense at all": perhaps you should consult this question on CV. – Silverfish May 31 '15 at 8:39
• Speculative rants are not considered appropriate answers on SE sites. They may be amusing--and may contain some truths, as I believe this one does--but in the end they die sad deaths, as in the bleak assessment of man's life in Macbeth, Act V, scene 5, lines 26-28. – whuber Jun 2 '15 at 21:56