So this is sort of a general/basic, likely dumb question. I'm hoping to get a general idea, to better guide what I search/read. How do these terms relate to each other. I know with Bayesian theory, you use priors to inform the model, where in frequentism you're just using the variables you have measured to build the model. Do both of these approaches use deterministic and stochastic models? Can you have a frequentist stochastic model?


Statistics, in general, is about learning from the data, that is thought as intrinsically random. There’s no truly deterministic models in statistics. Even when statistician does build a model that is based on some mechanistic model of the phenomenon of interest, she would additionally account for the noise in the data. The name “frequentism” itself is about thinking of probabilities as of frequencies. Bayesian approach is even more extreme, assuming everything to be a random variable. In both cases, probability theory serves as a foundation.

  • $\begingroup$ Is there a hierarchy? basically I'm trying to describe different modelling approaches to a disease/ epidemiology. Trying to fit the pieces together for some sort of narrative but some study authors state they use bayesian models, others stochastic, etc. Finding myself confused. $\endgroup$ – Jamie Imada Oct 13 '20 at 20:29
  • $\begingroup$ @JamieImada "stochastic" = "probabilistic" model, every Bayesian model is probabilistic, what is special about Bayesian ones is that they use priors. $\endgroup$ – Tim Oct 15 '20 at 18:08

Frequentist statistics are optimal methods. Maximum likelihood-based statistics are optimal methods. Bayesian statistics are optimal methods. They are each optimal at different things.

Each method is very good at solving certain types of problems. For some problems, the differences are minimal enough in practice that the differences are interpretive. There are problems where the differences can result in catastrophic outcomes. I am writing a paper on a real-world case right now.

I cannot use my cell phone as a surfboard. That is a feature, not a bug. My cell phone fits in my pocket, so standing and balancing on it is ill-advised. That is a feature. However, that is a bad fit for surfing. My cell phone is very dense. That allows it to have a wide range of functionality. It will also sink into the ocean. Also, bad for surfing.

You should choose your method based on the problems you are trying to solve. Do you need a surfboard or a phone?

Fisher's method of maximum likelihood, as originally built but not necessarily practiced today, was designed to help you decide if something is wrong. It was designed to give you new knowledge. However, that is it. Now you know something you did not.

In its original form, it produced a p-value. The smaller the p-value, the more likely it is that your hypothesis is false. Although that is not what a p-value formally is. Instead, it is the chance of seeing the specific data you saw, or something more extreme, if your null hypothesis is true, from chance alone. In Fisher's thinking, a p-value is the weight of the evidence against your null hypothesis.

The primary alternative null hypothesis method, there are others, such as the method of moments, is Pearson and Neyman's Frequentist theory. They realized there could be two problems with Fisher's thinking.

The first is that false negatives may be as important as false positives in real-world applications. The second, and it could be argued that this is minor, is that the maximum likelihood estimator is a biased estimator for most problems. Bias is a bit different from consistency. A consistent estimator eventually lands on the target, and maximum likelihood is usually consistent. However, bias could be thought about as a tendency to have more errors on one side of the target than on the other, but systematically.

Frequentist estimators carry two guarantees. First, they are, on average, perfectly accurate. Note the phrase "on average." Second, they provide a guaranteed level, upon infinite repetition, of false positives. For example, if you set $\alpha=.05$, then you are assured that if you infinitely repeat your experiment, no more than five percent of your positives will be false ones. In practice, you are setting the level or frequency for which you will be made a fool of, possibly in public, off incorrectly accepting a false positive. Likewise, they control for power, so they also provide a minimum level of protection against false negatives.

Given all these bonuses, why use Fisher's method at all? Because Fisher's method is the solution, that is the maximum likelihood answer. When Pearson and Neyman's method does not generate the same solution, then you are using a method of mediocre likelihood.

Fisher's method is the best fit to the facts, in some sense, but it is often biased and cannot protect you from being a fool too often.

Without getting into Bayesian methods, there is already at least a small dichotomy between the two. Do you need the best answer? Do you need a tool with a certain guaranteed level of internal reliability, even if the estimate may not be as good in the sense of precision? Do false negatives or positives matter? Do you need to make a decision with it, or are the inference and knowledge enough?

Are you going to set a cutoff of $\alpha$ or let the p-value stand on its own?

Except for meta-analysis, which is sort of like applying a prior probability to a new data set, all inference in the two main null hypothesis methods comes from the data alone. Both methods are built around the idea of a default piece of knowledge or a default action. That default is the null hypothesis.

In null hypothesis methods, the job of the data is to prove your default wrong.

There can be a material impact from this. Probabilities in these methodologies are conditioned on two things, the null hypothesis and the loss function. Because the sampling distribution of the mean and the median do not match, they imply different frequencies. That is true for any statistic. So when you perform a null hypothesis test, you are first conditioning the data on the null being a statement of fact and then the probabilities on the choice of statistics.

If the null is not a statement of fact, you do not know the frequencies. If you change your loss function, then you also change your frequencies. If you change your null hypothesis, you change your frequencies. That can matter.

The primary alternative family of statistics is the Bayesian one. In practice, it is one unified method, but there is more than one axiomatization of it. Each set of axioms can generate slightly different implications. That can also matter in the area of modeling if your goals are a bit more arcane than simple estimation.

All Bayesian methods are subjective, but so are the non-Bayesian ones as well. The difference is that Bayesian methods make the subjectivity open and available for criticism. Still, there is one element that makes Bayesian methods subjective in a way that Frequentist methods are not, except meta-analysis. That is the prior.

In Bayesian theory, everybody should have their own prior beliefs. Those beliefs could be quite unique. As such, every researcher seeing the same data should produce different results. It may be that those differences are out at the 32nd decimal place, but it is still different.

The goal of Bayesian methods is to improve your beliefs from data. As such, it can be difficult to work with since, in the scientific literature, you are reporting the results to someone other than yourself. They might not share your prior beliefs. In practice, that implies you have a lot more work to do to explain yourself.

Furthermore, the prior isn't something to avoid or play games with so you do not have to deal with it. Bayesian methods insist you quantify what you think you know or do not know. When reporting, however, you may want to attempt to construct your opponents' priors and work with them. If your reporting is still valid under very adverse prior assumptions, then they need to change their minds.

A prior is a probability distribution of $\theta$ and is based on information that was not part of your experiment. It could be from other experiments. It could be from professional experience. It could be very vague because you don't really know much. The likelihood is also a function about $\theta$ but it is based on your observed data. It isn't a distribution because it need not sum to one. The denominator normalizes the product of the two so that a new density (or mass) is created that sums to one.

This final distribution quantifies any remaining uncertainty about your question at hand.

So why use it?

Because it has several advantages.

First, all Bayesian estimators are admissible. That is to say, they are intrinsically the least risky way to calculate an estimator. A Bayesian estimator cannot be first-order stochastically dominated by another type of estimator. Other estimators can use as much information as a Bayesian estimator, but never more. There can be gigantic performance differences between a Bayesian and Frequentist estimator.

Second, Frequentist estimators can produce results that are physically impossible to be true. Usually, that does not matter, but it can. A trivial case would be where a confidence interval is over the range of $[-2,10]$ for a calorie count for food. Negative calories are impossible, but that is okay. As long as the overall interval covers the true parameter $1-\alpha$ percent of the time, it doesn't matter if negative calories could never happen. It can happen that a slope or a sample mean could be in an a priori impossible area.

Third, Bayesian hypothesis statements are positive and not negative. A null hypothesis statement goes like this: if the null is true, then the probability of seeing this result or one more extreme is $<p$, except for exact tests where it is an equality. Bayesian methods can have any countable number of hypotheses. Let us imagine a case with three hypotheses. We might be concerned whether $\theta$ is negative because that would have different implications from when it is positive. We might also be concerned that $0\le\theta<2$ because a small effect may imply something different from a large positive effect. The third hypothesis would be that $\theta\ge{2}$.

Frequentist tests condition on the null, so you could think of them as being $\Pr(X|\theta\le{k})$ where $X$ is your data and $\theta$ is your parameter. You condition your data on the hypothesis. Bayesian methods test, instead, $\Pr(\theta\le{k}|X)$. That is probably what you are actually interested in. You condition your hypothesis on the data. You probably never have been concerned about whether you are getting chance results if the hypothesis is true. You probably are more concerned about whether or not the hypothesis is true. Chance effects are another question and one you cannot control for in either method.

All statistical models are stochastic. Deterministic models, when using these methods, are really stochastic models in that they have admitted that there is at least one variable that could impact the outcome of an experiment that is not being observed.

A traditional deterministic model might be that $y=mx+b$. It stops being deterministic when you write it as $y=mx+b+\varepsilon,\varepsilon~\mathcal{N}(0,\sigma^2)$.

There is one slight technical difference between Bayesian and Frequentist models. Bayesian models are generative models, whereas Frequentist models are sampling-based models. A better Bayesian model fits the data generation function better even if it does not fit the data as well. A better Frequentist model could use different variables but do a better job at fitting the data.

If all of the assumptions of ordinary least squares regression holds, and the Bayesian statistician is using informative proper priors, then it is perfectly certain that the Frequentist least squares model will be a better fit to the observed data. That should be true for any data set. Nonetheless, the theorem that shows Bayesian methods are admissible still holds. While Frequentist methods are admissible when they either match a Bayesian estimator in every sample, or at the limit, they are not admissible in the presence of a proper prior.

It all boils down to fitness to purpose and why and for whom you need to use a method. There will be a learning curve in using a Bayesian versus a Frequentist model.

In a Bayesian model, the parameters are the random variables, not the data. Future data is random, but historical data is not. It is a statement of fact. In a Frequentist model, parameters are fixed but unknown quantities, and the data, both past, and future are randomly drawn.

In the subjective school of Bayesian probability, when you perform an experiment, Nature draws the values of your parameters from your prior. In the objective school, the parameters are fixed points but the prior quantifies your uncertainty. They are mathematically equivalent statements. They are not philosophically equivalent.

  • $\begingroup$ A great exposition! I have a quibble though: A prior is a probability distribution of $\theta$ and is based on information that was not part of your experiment. <...>The likelihood is also a function about $\theta$ but it is based on your observed data. In many practical problems, the shape of the likelihood (the assumed distribution) is based on information that was not part of your experiment, so in some sense it is as subjective as the prior. I think Andrew Gelman likes to stress that. $\endgroup$ – Richard Hardy Jan 30 at 19:57
  • $\begingroup$ @RichardHardy if we go to the magical world of mathematicians, that should never happen because Bayesian models are generative. Nature alone should determine the form of the likelihood. In the real world, judgment is critical and models may not be generative because the needed variable isn't observed or in the set. There is a subjectivity present and a level of skill too. An engineering intern taking a soil sample for a massive civil engineering problem may view it in a very different manner than a person with three decades of engineering working on this particular problem. $\endgroup$ – Dave Harris Jan 30 at 23:47
  • 1
    $\begingroup$ I think the problem is not with the true data generating process but with our inability to guess it right. So we are guessing the shape of the likelihood subjectively, and we can make a mistake there. If we do specification testing, we can detect particularly poor guesses and replace them with better ones, but some amount of subjectivity remains e.g. regarding the set of likelihoods that we try out. $\endgroup$ – Richard Hardy Jan 31 at 7:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.