Why is statistics useful when many things that matter are one shot things?

Question

I don't know if it's just me, but I am very skeptical of statistics in general. I can understand it in dice games, poker games, etc. Very small, simple, mostly self-contained repeated games are fine. For example, a coin landing on its edge is small enough to accept the probability that landing heads or tails is ~50%.

Playing a $10 game of poker aiming for a 95% win is fine. But what if your entire life savings + more is dependent on you hitting a win or not? How would knowing that you'd win in 95% of the time in that situation will help me at all? Expected value doesn't help much there.

Other examples include a life-threatening surgery. How does that help knowing that it is 51% survival rate versus 99% survival rate given existing data? In both cases, I don't think it will matter to me what the doctor tells me, and I would go for it. If actual data is 75%, he might as well tell me (barring ethics and law), that there is a 99.99999% chance of survival so I'd feel better. In other words, existing data doesn't matter except binomially. Even then, it doesn't matter if there is a 99.99999% survival rate, if I end up dying from it.

Also, earthquake probability. It doesn't matter if a strong earthquake happened every x (where x > 100) years on average. I have no idea if an earthquake will happen ever in my lifetime. So why is it even useful information?

A less serious example, say, 100% of the places I've been to that I love are in the Americas, indifferent to 100% of the places I've been to in Europe, and hate 100% of the places that I have been to in Asia. Now, that by no means mean that I wouldn't find a place that I love in Asia on my next trip or hate in Europe or indifferent in America, just by the very nature that the statistics doesn't capture all of the information I need, and I probably can never capture all of the information I need, even if I have traveled to over x% of all of those continents. Just because there are unknowns in the 1-x% of those continents that I haven't been to. (Feel free to replace the 100% with any other percentage).

I understand that there is no way to brute force everything and that you have to rely on statistics in many situations, but how can we believe that statistics are helpful in our one shot situation, especially when statistics basically do not extrapolate to outlier events?

Any insights to get over my skepticism of statistics?

Answer 1

First I think that you may be confusing "statistics" meaning a collection of numbers or other facts describing a group or situation, and "statistics" meaning the science of using data and information to understand the world in the face of variation (others may be able to improve on my definitions). Statisticians use both senses of the word, so it is not surprising when people mix them up.

Statistics (the science) is a lot about choosing strategies and choosing the best strategy even if we only get to apply it once. Some times when I (and others) teach probability we use the classic Monty Hall problem (3 doors, 2 goats, 1 car) to motivate it and we show how we can estimate probabilities by playing the game a bunch of times (not for prizes) and we can see that the "switch" strategy wins 2/3 of the time and the "stay" strategy only wins 1/3 of the time. Now if we had the opportunity to play the game a single time we would know some things about which strategy gives a better chance of winning.

The surgery example is similar, you will only have the surgery (or not have the surgery) once, but don't you want to know which strategy benifits more people? If your choices are surgery with some chance greater than 0% of survival or no surgery and 0% of survival, then yes there is little difference between the surgery having 51% survival and 99.9% survival. But what if there are other options as well, you can choose between surgery, doing nothing (which has 25% survival) or a change of diet and exercise which has 75% survival (but requires effort on your part), now wouldn't you care about if the surgery option has 51% vs. 99% survival?

Also consider the doctor, he will be doing more than just your surgery. If surgery has 99.9% survival then he has no reason to consider alternatives, but if it only has 51% survival then while it may be the best choice today, he should be looking for other alternatives that increase that survival. Yes even with 90% survival he will loose some patients, but which strategy gives him the best chance of saving the most patients?

This morning I wore my seat belt while driving (my usual strategy), but did not get in any accidents, so was my strategy a waste of time? If I knew when I would get in an accident then I could save time by only putting on the seat belt on those occasions and not on others. But I don't know when I will be in an accident so I will stick with my wear the seat belt strategy because I believe it will give me the best chance if I ever am in an accident even if that means wasting a bit of time and effort in the high percentage (hopefully 100%) of times that there is no accident.

Answer 2

Just because you don't use statistics in your daily life does not mean that the field does not directly affect you. When you are at the doctor and they recommend one treatment over the other, you can bet that behind that recommendation was many clinical trials that used statistics to interpret the results of their experiments.

It turns out that the concept of expected value is also very useful even if you do not personally use the concept. Your example of betting your life savings fails to take into account how risk adverse you are. Other situations might find yourself less risk adverse, or where there are not catastrophic outcomes. Business, finance, actuarial contexts and others are examples of this. Perhaps you are issuing home insurance policy - then all of the sudden knowing the probability of an earthquake occurring within some specified period of time matters a great deal.

In the end statistics is a great way to deal with uncertainty. Your last example you made up some data about places you like to travel and claimed that statistics will say that you will never find a place in Asia that you like. This is just wrong. Of course this data will make you believe that Asia is less likely to have a place you like, but you can set your prior belief to be whatever you like, and statistics will tell you how to update your belief given the new data. Furthermore, it allows you to do modify your belief in a principled way that will allow you to act rationally in the presence of uncertainty.

Answer 3

I myself have the same doubts about the usefulness of probability, and statistics, when it comes to taking decision about a single event. In my opinion, knowing the probability, real or estimated, is extremely important when the objective is estimating outcomes of samples, be they a single event repeated a number of times or a sample drown from a certain population. In short, knowing the probability makes more sense for the casino who, based on probability calculations can put the rules that guarantee he would win in the long run (after many plays) and not for a gambler who pretends to play one time, so he would won or loose (these are the outcomes when the experiment is run a single time). It’s also important for the generals who contemplate sending their solders to a battle with the risk (probability) of losing 10% of them, but not for a certain solder (say, John) who is only going to die or survive. There are so many examples like these in real life.

The point I want to make is that, Probability and Statistics, not only are useful in real life but, more precisely, they are a tool for all modern scientific researches and decision making rules. However, it’s not correct to say that rationality implies relying on the probability of a single event, without the intention or the possibility of repeating it, for estimating the outcome. The tendency of the probability to influence a certain individual’s decision, based on her or his degree of risk aversion, is obviously subjective. Risk avert and risk lover have different attitudes (decisions) toward the same lottery (the same expected value).

Answer 4

The world is stochastic not deterministic. If it were deterministic the physicists would be ruling the world and statisticians would be out of a job. But the reality is that statisticians are in high demand in almost every discipline. That is not to say that there isn't a place for physics and other sciences but statistics works hand in hand with science and is the basis for many scientific discoveries.

Enough chatter and down to specifics. I have worked the last 17 years in the medical industry, first in medical devices, then pharmaceuticals, and now general medical research. Drugs and medical devices that improve quality of life and often save or extend life are developed and approved in this country and around the world on a regular basis. In the US approval requires evidence of safety and efficacy before the FDA will allow a drug or medical device to be marketed. Evidence to the FDA comes from clinical trials in phases. All the clinical trials require valid statistical design and analysis methods. Nothing is perfect. Drugs work well for some people while others may not respond or will have adverse events (bad reactions that can cause illness or death). The trials separate out the ineffective drugs from the effective. Most drugs fail and there is often a ten year cycle from early stage development to end of phase III with approval and marketing at the end of the trial. Postmarket surveillance which also requires statistics is then applied to make sure that the drug works well enough for the general population. Sometimes the general population that the drug is approved for is a less restrictive group than the patients that were eligible for the clinical trials. So sometimes drugs do turn out to be dangerous and get pulled from the market. Statistics helps in all aspects of drug safety.

Statistics is not perfect. We live with some mistakes due to randomness and uncertainty. But it is controlled and our lives are better and errors are reduced from what they would be had statistical science not been involved.

Answer 5

I skeptical of statistics for the following reasons.

1. I am convinced anybody without a graduate degree in statistics has no clue what they are doing. Unf. there are millions of people across the world doing research without a graduate degree in statistics. I was an undergraduate math major at the Univeristy of Maryland College, Park. I took 4 400 level math classes. All teachers did was teach you how to calculate stuff. Nobody taught me how to make sense of anything or do any statistical analysis except for hypothesis testing, which makes no sense for 2 reasons.
   1. For every hypothesis test I was taught, I had to make assumptions beforehand. Nobody taught me which assumption(s) I had to start with. 2. P values make no sense logically. A graduate degree in statistics might teach you what a p value actually is. However, I am convinced no undergraduate knows how to use it. The undergraduate definition assumes a probability of something that depends on the hypothesis being correct. Logically, the definition makes no sense at all. Even worse, NOBODY has ever told me where the probability comes from. I have actually emailed almost my whole math department(more than 200 people) if somebody could give me an answer. The most popular and only responses were "one would have to ASSUME the error rates for the probability" (When I asked people how this was done, they all answered me "from previous experiments", I could not get anything more specific from anybody), "It's just the way it is", and "it is completely random".
   The same thing happened when I googled what the signifigance of a p value is. It leads me to the conclusion...

2. Even a sig. number of math and statistics professors have no clue what the logic behind statistics is. I don't expect people to have in depth knowledge. However, I have a feeling that even a sig. % of researches and professors do not understand any of the underlying logic behind statistics.

3. Statistical error is not the same thing as actual error. Because people like to use statistics to derive estimates for things that are humongous, people like to use statistical error to "mask" the fact that they have no clue what the actual error.

4. People use small samples for big populations because statistical theory tells them they can. I learned from one of my college courses, that people like to use data that is an estimate from about 30 schools in the country to show that there are few violent incidents in schools in the whole country. There are about 100,000 schools. That sounds insane. A whole popular movement is based off of about 30 schools in the whole country.

5. People like to make the burden of proof statistical. The Higgs Bossom was never discovered. It was discovered statistically, but that doesn't mean anything. Something being discovered purely statistically is useless because nobody knows the accuracy of statistics.

6. People like to use statistics to make important desicions. Statistics can be used as a guide, but nobody knows how accurate it really is. Just because a problem seems impossible to solve does not mean that statistics is the next best thing. The fact that DNA testing is based off of statistic gives me the chills. Can I be given the death penalty soley because of statistics? Could a murderer be released from jail soley because of statistics?

I believe statistics can be useful, but only if it is not used as the conclusion. I believe statistics can tell us what some of the possibilities are. Then logic, not statistical logic should be used to prove which possibility(s) is correct.

Answer 6

The long and the short of it is that probability is the unique generalization of ordinary true/false logic to degrees of belief between 0 and 1. This is the so-called logical Bayesian interpretation of probability, originated by R.T. Cox and later championed by E.T. Jaynes.

Furthermore under weak assumptions it can be shown that the right way to order uncertain outcomes by preference is to order them by expected utility, with the expected taken with respect to the probability distribution over outcomes.

See Robert Clemen, "Making Hard Decisions", for an introduction and exposition on applied decision analysis which is based on Bayesian probability and expected utility.

You are absolutely right to be skeptical about conventional frequentist statistics; by the design of its inventors (R.A. Fisher, J. Neyman, E. Pearson) it is limited to repetitive events. But many everyday problems don't involve repetitive events. What to do? The typical approach is some combination of forcing square pegs into round holes, and moving the goalposts. Shameful, really.