Why is statistics useful when many things that matter are one shot things?

Question

I don't know if it's just me, but I am very skeptical of statistics in general. I can understand it in dice games, poker games, etc. Very small, simple, mostly self-contained repeated games are fine. For example, a coin landing on its edge is small enough to accept the probability that landing heads or tails is ~50%.

Playing a $10 game of poker aiming for a 95% win is fine. But what if your entire life savings + more is dependent on you hitting a win or not? How would knowing that you'd win in 95% of the time in that situation will help me at all? Expected value doesn't help much there.

Other examples include a life-threatening surgery. How does that help knowing that it is 51% survival rate versus 99% survival rate given existing data? In both cases, I don't think it will matter to me what the doctor tells me, and I would go for it. If actual data is 75%, he might as well tell me (barring ethics and law), that there is a 99.99999% chance of survival so I'd feel better. In other words, existing data doesn't matter except binomially. Even then, it doesn't matter if there is a 99.99999% survival rate, if I end up dying from it.

Also, earthquake probability. It doesn't matter if a strong earthquake happened every x (where x > 100) years on average. I have no idea if an earthquake will happen ever in my lifetime. So why is it even useful information?

A less serious example, say, 100% of the places I've been to that I love are in the Americas, indifferent to 100% of the places I've been to in Europe, and hate 100% of the places that I have been to in Asia. Now, that by no means mean that I wouldn't find a place that I love in Asia on my next trip or hate in Europe or indifferent in America, just by the very nature that the statistics doesn't capture all of the information I need, and I probably can never capture all of the information I need, even if I have traveled to over x% of all of those continents. Just because there are unknowns in the 1-x% of those continents that I haven't been to. (Feel free to replace the 100% with any other percentage).

I understand that there is no way to brute force everything and that you have to rely on statistics in many situations, but how can we believe that statistics are helpful in our one shot situation, especially when statistics basically do not extrapolate to outlier events?

Any insights to get over my skepticism of statistics?

Answer 1

First I think that you may be confusing "statistics" meaning a collection of numbers or other facts describing a group or situation, and "statistics" meaning the science of using data and information to understand the world in the face of variation (others may be able to improve on my definitions). Statisticians use both senses of the word, so it is not surprising when people mix them up.

Statistics (the science) is a lot about choosing strategies and choosing the best strategy even if we only get to apply it once. Some times when I (and others) teach probability we use the classic Monty Hall problem (3 doors, 2 goats, 1 car) to motivate it and we show how we can estimate probabilities by playing the game a bunch of times (not for prizes) and we can see that the "switch" strategy wins 2/3 of the time and the "stay" strategy only wins 1/3 of the time. Now if we had the opportunity to play the game a single time we would know some things about which strategy gives a better chance of winning.

The surgery example is similar, you will only have the surgery (or not have the surgery) once, but don't you want to know which strategy benifits more people? If your choices are surgery with some chance greater than 0% of survival or no surgery and 0% of survival, then yes there is little difference between the surgery having 51% survival and 99.9% survival. But what if there are other options as well, you can choose between surgery, doing nothing (which has 25% survival) or a change of diet and exercise which has 75% survival (but requires effort on your part), no wouldn't you care about if the surgery option has 51% vs. 99% survival?

Also consider the doctor, he will be doing more than just your surgery. If surgery has 99.9% survival then he has no reason to consider alternatives, but if it only has 51% survival then while it may be the best choice today, he should be looking for other alternatives that increase that survival. Yes even with 90% survival he will loose some patients, but which strategy gives him the best chance of saving the most patients?

This morning I wore my seat belt while driving (my usual strategy), but did not get in any accidents, so was my strategy a waste of time? If I knew when I would get in an accident then I could save time by only putting on the seat belt on those occasions and not on others. But I don't know when I will be in an accident so I will stick with my wear the seat belt strategy because I believe it will give me the best chance if I ever am in an accident even if that means wasting a bit of time and effort in the high percentage (hopefully 100%) of times that there is no accident.

Answer 2

Just because you don't use statistics in your daily life does not mean that the field does not directly affect you. When you are at the doctor and they recommend one treatment over the other, you can bet that behind that recommendation was many clinical trials that used statistics to interpret the results of their experiments.

It turns out that the concept of expected value is also very useful even if you do not personally use the concept. Your example of betting your life savings fails to take into account how risk adverse you are. Other situations might find yourself less risk adverse, or where there are not catastrophic outcomes. Business, finance, actuarial contexts and others are examples of this. Perhaps you are issuing home insurance policy - then all of the sudden knowing the probability of an earthquake occurring within some specified period of time matters a great deal.

In the end statistics is a great way to deal with uncertainty. Your last example you made up some data about places you like to travel and claimed that statistics will say that you will never find a place in Asia that you like. This is just wrong. Of course this data will make you believe that Asia is less likely to have a place you like, but you can set your prior belief to be whatever you like, and statistics will tell you how to update your belief given the new data. Furthermore, it allows you to do modify your belief in a principled way that will allow you to act rationally in the presence of uncertainty.

Answer 3

The world is stochastic not deterministic. If it were deterministic the physicists would be ruling the world and statisticians would be out of a job. But the reality is that statisticians are in high demand in almost every discipline. That is not to say that there isn't a place for physics and other sciences but statistics works hand in hand with science and is the basis for many scientific discoveries.

Enough chatter and down to specifics. I have worked the last 17 years in the medical industry, first in medical devices, then pharmaceuticals, and now general medical research. Drugs and medical devices that improve quality of life and often save or extend life are developed and approved in this country and around the world on a regular basis. In the US approval requires evidence of safety and efficacy before the FDA will allow a drug or medical device to be marketed. Evidence to the FDA comes from clinical trials in phases. All the clinical trials require valid statistical design and analysis methods. Nothing is perfect. Drugs work well for some people while others may not respond or will have adverse events (bad reactions that can cause illness or death). The trials separate out the ineffective drugs from the effective. Most drugs fail and there is often a ten year cycle from early stage development to end of phase III with approval and marketing at the end of the trial. Postmarket surveillance which also requires statistics is then applied to make sure that the drug works well enough for the general population. Sometimes the general population that the drug is approved for is a less restrictive group than the patients that were eligible for the clinical trials. So sometimes drugs do turn out to be dangerous and get pulled from the market. Statistics helps in all aspects of drug safety.

Statistics is not perfect. We live with some mistakes due to randomness and uncertainty. But it is controlled and our lives are better and errors are reduced from what they would be had statistical science not been involved.

Answer 4

The long and the short of it is that probability is the unique generalization of ordinary true/false logic to degrees of belief between 0 and 1. This is the so-called logical Bayesian interpretation of probability, originated by R.T. Cox and later championed by E.T. Jaynes.

Furthermore under weak assumptions it can be shown that the right way to order uncertain outcomes by preference is to order them by expected utility, with the expected taken with respect to the probability distribution over outcomes.

See Robert Clemen, "Making Hard Decisions", for an introduction and exposition on applied decision analysis which is based on Bayesian probability and expected utility.

You are absolutely right to be skeptical about conventional frequentist statistics; by the design of its inventors (R.A. Fisher, J. Neyman, E. Pearson) it is limited to repetitive events. But many everyday problems don't involve repetitive events. What to do? The typical approach is some combination of forcing square pegs into round holes, and moving the goalposts. Shameful, really.