# Why is statistics useful when many things that matter are one shot things?

I don't know if it's just me, but I am very skeptical of statistics in general. I can understand it in dice games, poker games, etc. Very small, simple, mostly self-contained repeated games are fine. For example, a coin landing on its edge is small enough to accept the probability that landing heads or tails is ~50%.

Playing a $10 game of poker aiming for a 95% win is fine. But what if your entire life savings + more is dependent on you hitting a win or not? How would knowing that you'd win in 95% of the time in that situation will help me at all? Expected value doesn't help much there. Other examples include a life-threatening surgery. How does that help knowing that it is 51% survival rate versus 99% survival rate given existing data? In both cases, I don't think it will matter to me what the doctor tells me, and I would go for it. If actual data is 75%, he might as well tell me (barring ethics and law), that there is a 99.99999% chance of survival so I'd feel better. In other words, existing data doesn't matter except binomially. Even then, it doesn't matter if there is a 99.99999% survival rate, if I end up dying from it. Also, earthquake probability. It doesn't matter if a strong earthquake happened every x (where x > 100) years on average. I have no idea if an earthquake will happen ever in my lifetime. So why is it even useful information? A less serious example, say, 100% of the places I've been to that I love are in the Americas, indifferent to 100% of the places I've been to in Europe, and hate 100% of the places that I have been to in Asia. Now, that by no means mean that I wouldn't find a place that I love in Asia on my next trip or hate in Europe or indifferent in America, just by the very nature that the statistics doesn't capture all of the information I need, and I probably can never capture all of the information I need, even if I have traveled to over x% of all of those continents. Just because there are unknowns in the 1-x% of those continents that I haven't been to. (Feel free to replace the 100% with any other percentage). I understand that there is no way to brute force everything and that you have to rely on statistics in many situations, but how can we believe that statistics are helpful in our one shot situation, especially when statistics basically do not extrapolate to outlier events? Any insights to get over my skepticism of statistics? - (+1) Welcome to our site! It isn't just you: this is a deep question that goes to the foundations of statistics. – whuber Sep 7 '12 at 22:06 The "life savings" example mixes separate issues. In economics, a common model for rational risk aversion is to maximize expected utility, not expected money, where utility is typically a concave (sublinear) function like log(money). This means losses cost more than gains of the same size, and this effect is larger for larger changes. This is very different from not believing there is any difference between$50\%$and$99\%\$, which leads to inconsistent and irrational behavior. – Douglas Zare Sep 8 '12 at 7:28
@DouglasZare this sounds like a very interesting area. Can you provide an introductory article to the topic of individual risk aversion regarding live savings ? – steffen Sep 8 '12 at 8:37
@steffen: This material is covered in many basic economics texts. The theory of expected utility maximization is viewed by many as too simple, and insufficient to explain many phenomena, but it is an important starting point to understand before moving on to ideas such as prospect theory. Something which is easily explained by expected utility maximization instead of expected money maximization should not be viewed as a failure of probability theory. en.wikipedia.org/wiki/Expected_utility_hypothesis – Douglas Zare Sep 8 '12 at 8:46

First I think that you may be confusing "statistics" meaning a collection of numbers or other facts describing a group or situation, and "statistics" meaning the science of using data and information to understand the world in the face of variation (others may be able to improve on my definitions). Statisticians use both senses of the word, so it is not surprising when people mix them up.

Statistics (the science) is a lot about choosing strategies and choosing the best strategy even if we only get to apply it once. Some times when I (and others) teach probability we use the classic Monty Hall problem (3 doors, 2 goats, 1 car) to motivate it and we show how we can estimate probabilities by playing the game a bunch of times (not for prizes) and we can see that the "switch" strategy wins 2/3 of the time and the "stay" strategy only wins 1/3 of the time. Now if we had the opportunity to play the game a single time we would know some things about which strategy gives a better chance of winning.

The surgery example is similar, you will only have the surgery (or not have the surgery) once, but don't you want to know which strategy benifits more people? If your choices are surgery with some chance greater than 0% of survival or no surgery and 0% of survival, then yes there is little difference between the surgery having 51% survival and 99.9% survival. But what if there are other options as well, you can choose between surgery, doing nothing (which has 25% survival) or a change of diet and exercise which has 75% survival (but requires effort on your part), no wouldn't you care about if the surgery option has 51% vs. 99% survival?

Also consider the doctor, he will be doing more than just your surgery. If surgery has 99.9% survival then he has no reason to consider alternatives, but if it only has 51% survival then while it may be the best choice today, he should be looking for other alternatives that increase that survival. Yes even with 90% survival he will loose some patients, but which strategy gives him the best chance of saving the most patients?

This morning I wore my seat belt while driving (my usual strategy), but did not get in any accidents, so was my strategy a waste of time? If I knew when I would get in an accident then I could save time by only putting on the seat belt on those occasions and not on others. But I don't know when I will be in an accident so I will stick with my wear the seat belt strategy because I believe it will give me the best chance if I ever am in an accident even if that means wasting a bit of time and effort in the high percentage (hopefully 100%) of times that there is no accident.

-
 +1 Greg Good post! I was writing mine at the same time as you. We may overlap a little but I think we both had things to say that were right on the mark and did not overlap. I am not sure what the OP thinks statistics is. It is nice that you gave him the benefit of the doubt. I took a more angry approach to it. – Michael Chernick Sep 7 '12 at 21:55 Hi Greg, I liked your answer, but can I reason it like this: statistics (the science) is itself a statistic, it works for x% of the time, (possibly high x), but there are the 1-x% unknown/random factors that we always need to be aware of. Given that we can model the unknown in any # (possibly infinite) of ways, we will never know x. Hopefully these outliers will never happen, but we should always be mindful and err on the conservative, especially if the event is catastrophic (ie. asteroids, financial products, nuclear accidents for society and car accidents for personal). Does this make sense? – statskeptic Sep 10 '12 at 18:07 @statskeptic, what you say applies to all areas, not just statistics. In fact is applies less to true statistics than other fields because when statistics is done right the assumptions are clear. Most times that statistics has failed it has not been the techniques, but that they were applied incorrectly. In any field that involves uncertainty (which is pretty much anything other than religion or pure math, and even they have some) you can have an answer that is either wrong, useless, or uses statistics. – Greg Snow Sep 11 '12 at 16:55

Just because you don't use statistics in your daily life does not mean that the field does not directly affect you. When you are at the doctor and they recommend one treatment over the other, you can bet that behind that recommendation was many clinical trials that used statistics to interpret the results of their experiments.

It turns out that the concept of expected value is also very useful even if you do not personally use the concept. Your example of betting your life savings fails to take into account how risk adverse you are. Other situations might find yourself less risk adverse, or where there are not catastrophic outcomes. Business, finance, actuarial contexts and others are examples of this. Perhaps you are issuing home insurance policy - then all of the sudden knowing the probability of an earthquake occurring within some specified period of time matters a great deal.

In the end statistics is a great way to deal with uncertainty. Your last example you made up some data about places you like to travel and claimed that statistics will say that you will never find a place in Asia that you like. This is just wrong. Of course this data will make you believe that Asia is less likely to have a place you like, but you can set your prior belief to be whatever you like, and statistics will tell you how to update your belief given the new data. Furthermore, it allows you to do modify your belief in a principled way that will allow you to act rationally in the presence of uncertainty.

-
The traveling example was just a made up one, but the idea is that statistics does not capture the unknown. Your example of business contexts made me thought of the example of insurance companies of WTC probably estimated the cost/benefit of insuring the building without taking into account of planes destroying the building, but yet it is the one that mattered most. – statskeptic Sep 7 '12 at 22:11
+1 @jjund3 for addressing the OPs specific questions and for intermingling Bayesian and frequentist staistics without any conflict. – Michael Chernick Sep 7 '12 at 22:55
@statskeptic Your point that statistics cannot account for all possible uncertainties is a good one. But it doesn't have to be complete and perfect to be useful. We do have knowledge about terrorists. Prior to 9/11 we did have examples of terrorists going on suicide missions and we had experience with highjacking of planes. The information could have been pieced together to determine that crashing aplane into the World Trade Center was a possibility although we would probably have assessed it as a remote possibility. – Michael Chernick Sep 7 '12 at 23:02
We knew the World Trade Center was a favorite terrist target. It had been attacked once before with a bomb set off in the basement. The fact that the bomb was not strong enough to do the desired damage was at least a hint that the next time some very different method would be used. Of course as often said hind sight is 20-20. There are many examples where the unexpected or improbable happens. But not in the case of the Challanger disaster. There the Thiokol engineers even with limited data knew that there was some riks of a catastrophic failure due to O-rings failure at low temperature. – Michael Chernick Sep 7 '12 at 23:07
@statskeptic Your argument is very similar to Taleb's skepticism/ bashing of statistics in his book the Black Swan. I think many statisticians myself included have shot holes in his argument which basially says that statistics is useless because it cannot predict that rare and unthinkable event (9/11 in your example,the stock market crash in his). – Michael Chernick Sep 7 '12 at 23:19

The world is stochastic not deterministic. If it were deterministic the physicists would be ruling the world and statisticians would be out of a job. But the reality is that statisticians are in high demand in almost every discipline. That is not to say that there isn't a place for physics and other sciences but statistics works hand in hand with science and is the basis for many scientific discoveries.

Enough chatter and down to specifics. I have worked the last 17 years in the medical industry, first in medical devices, then pharmaceuticals, and now general medical research. Drugs and medical devices that improve quality of life and often save or extend life are developed and approved in this country and around the world on a regular basis. In the US approval requires evidence of safety and efficacy before the FDA will allow a drug or medical device to be marketed. Evidence to the FDA comes from clinical trials in phases. All the clinical trials require valid statistical design and analysis methods. Nothing is perfect. Drugs work well for some people while others may not respond or will have adverse events (bad reactions that can cause illness or death). The trials separate out the ineffective drugs from the effective. Most drugs fail and there is often a ten year cycle from early stage development to end of phase III with approval and marketing at the end of the trial. Postmarket surveillance which also requires statistics is then applied to make sure that the drug works well enough for the general population. Sometimes the general population that the drug is approved for is a less restrictive group than the patients that were eligible for the clinical trials. So sometimes drugs do turn out to be dangerous and get pulled from the market. Statistics helps in all aspects of drug safety.

Statistics is not perfect. We live with some mistakes due to randomness and uncertainty. But it is controlled and our lives are better and errors are reduced from what they would be had statistical science not been involved.

-
 Don't get me wrong. I understand there is statistics in everything, even physics with quantum mechanics is all about probability, and there isn't enough atoms to do computation without statistics. I just want to find out how to deal with the randomness and uncertainty which can influence my (or other people's) life more than any actual statistic or distribution. – statskeptic Sep 7 '12 at 22:00 Okay statskeptic so you are not confused. But why is it so hard to see how statistics improves your chance for success. Probability theory tells you the odds of winning games of chance. If you could use Thorpe's Beat the Dealer strategy in blackjack and you have a large bank of funds you can make a fortune in the long run. The MIT students proved it in Las Vegas even though the advantage in counting has been reduced by the use of mixing multiple decks. It is true. The casino knows that card counters are a threat. – Michael Chernick Sep 7 '12 at 22:16 They search for them and when they think they find one they throw him or her out of the casino no questions asked. – Michael Chernick Sep 7 '12 at 22:17 Also, please don't think that I'm trying to flame your profession. There are computers that are doing computation statistically to save power, and I respect that. I'm just trying to learn how people with much more knowledge than me in statistics deal with these questions. – statskeptic Sep 7 '12 at 22:19 @statskeptic I fyou saw my original post I apologize for my inital comments. They were rightfully edited out by a moderator. I think I misunderstood what you were trying to say. I hope we answered your question well and relieved some of your skepticism. – Michael Chernick Sep 7 '12 at 22:44

The long and the short of it is that probability is the unique generalization of ordinary true/false logic to degrees of belief between 0 and 1. This is the so-called logical Bayesian interpretation of probability, originated by R.T. Cox and later championed by E.T. Jaynes.

Furthermore under weak assumptions it can be shown that the right way to order uncertain outcomes by preference is to order them by expected utility, with the expected taken with respect to the probability distribution over outcomes.

See Robert Clemen, "Making Hard Decisions", for an introduction and exposition on applied decision analysis which is based on Bayesian probability and expected utility.

You are absolutely right to be skeptical about conventional frequentist statistics; by the design of its inventors (R.A. Fisher, J. Neyman, E. Pearson) it is limited to repetitive events. But many everyday problems don't involve repetitive events. What to do? The typical approach is some combination of forcing square pegs into round holes, and moving the goalposts. Shameful, really.

-
-1 In my opinion a very poor and unfair portrayal of frequentist statistics. I would not take such a negative view of the Bayesian approach. But the Bayesians (any camp) are not free from criticism. Is degree of belief to be the staple of inference? Is degree of belief subjective and personal, so that two people can give two different answers? What about the need for a prior distribution? How should it be picked? Lots of questions for any paradigm for inference. But aren't we past the stage of hostle bickering over the foundations? – Michael Chernick Sep 7 '12 at 22:05
There is more about the scientific method to unite us and say resoundingly that STAISTICS IS IMPORTANT when facing a skeptic. Instead you agree with the skeptic in order to take a cheap shot at frequentist methods! That is what is shameful. – Michael Chernick Sep 7 '12 at 22:06
@MichaelChernick: (1) to simply shout STATISTICS IS IMPORTANT is hardly an argument which will win over a skeptic. (2) Bayesian inference has the same relation to problem data as ordinary logic. That is, given some premises, you crank out a solution by applying laws of probability. The data (e.g. any prior distribution) are neither right nor wrong; they just are. Reasonable people disagree about prior distributions just as they might about any other problem data. – Robert Dodier Sep 7 '12 at 23:24
(ahem ... continuing) (3) If you cannot state all the data needed to reach a conclusion, then you can't apply Bayesian inference. If for some reason you really can't state a prior distribution, then you simply have to admit the problem isn't solvable by Bayesian inference. (4) If you think I've made some inaccurate statements about frequentist statistics, by all means, please give your best rebuttal. – Robert Dodier Sep 7 '12 at 23:33
Hug it out guys. – Brandon Bertelsen Sep 8 '12 at 11:54