# Subjectivity in Frequentist Statistics

I often hear the claim that Bayesian statistics can be highly subjective. The main argument being that inference depends on the choice of a prior (even though one could use the principle of indifference o maximum entropy to choose a prior). In comparison, the claim goes, frequentist statistics is in general more objective. How much truth is there in this statement?

Also, this makes me wonder:

1. What are concrete elements of frequentist statistics (if any) that can be particularly subjective and that are not present or are less important in Bayesian statistics?
2. Is subjectivity more prevalent in Bayesian than in frequentist statistics?
-
Certainly frequentist methods are much more subjective than advertised, but I would argue that Bayesian methods are more subjective still. And please don't convince yourself that invariance of your prior under a group or the specification of a MaxEnt prior are somehow "objective" - both types prior potentially express beliefs that I consider informative, and at any rate neither strategy is applicable in complete generality (e.g. I don't think there are invariant or MaxEnt priors on the space of CDFs, and any prior on this space assigns probability 1 to a topologically meager set). – guy Jul 14 '14 at 0:56
Bayesians start on the subjective ground, then data (hopefully) pulls them back into the objective reality. Frequentists start (or at least think that they do) from objective positions, but then they end up tainting the analysis with their subjective assumptions. – Aksakal Jul 14 '14 at 1:03
Bayesians both know and are up front with their assumptions. Frequentists are generally not. – Alexis Jul 14 '14 at 22:20

I often hear the claim that Bayesian statistics can be highly subjective.

So do I. But notice that there's a major ambiguity in calling something subjective.

Subjectivity (both senses)

Subjective can mean (at least) one of

1. depends on the idiosyncracies of the researcher
2. explicitly concerned with the state of knowledge of an individual

Bayesianism is subjective in the second sense because it is always offering a way to update beliefs represented by probability distributions by conditioning on information. (Note that whether those beliefs are beliefs that some subject actually has or just beliefs that a subject could have is irrelevant to deciding whether it is 'subjective'.)

The main argument being that inference depends on the choice of a prior

Actually, if a prior represents your personal belief about something then you almost certainly didn't choose it at any more than you chose most of your beliefs. And if it represents somebody's beliefs then it can be a more or less accurate representation of those beliefs, so ironically there will be a rather 'objective' fact about how well it represents them.

(even though one could use the principle of indifference o maximum entropy to choose a prior).

One could, though this doesn't tend to generalize very smoothly to continuous domains. Also, arguably it's impossible to be flat or 'indifferent' in all parameterisations at once (though I've never been quite sure why you'd want to be).

In comparison, the claim goes, frequentist statistics is in general more objective. How much truth is there in this statement?

So how might we evaluate this claim?

I suggest that in the second second sense of subjective: it's mostly correct. And in the first sense of subjective: it's probably false.

Frequentism as subjective (second sense)

Some historical detail is helpful to map the issues

For Neyman and Pearson there is only inductive behaviour not inductive inference and all statistical evaluation works with long run sampling properties of estimators. (Hence alpha and power analysis, but not p values). That's pretty unsubjective in both senses.

Indeed it's possible, and I think quite reasonable, to argue along these lines that Frequentism is actually not an inference framework at all but rather a collection of evaluation criteria for all possible inference procedures that emphasises their behaviour in repeated application. Simple examples would be consistency, unbiasedness, etc. This makes it obviously unsubjective in sense 2. However, it also risks being subjective in sense 1 when we have to decide what to do when those crteria do not apply (e.g. when there isn't an unbiased estimator to be had) or when they apply but contradict.

Fisher offered a less unsubjective Frequentism that is interesting. For Fisher, there is such a thing as inductive inference, in the sense that a subject, the scientist, makes inferences on the basis of a data analysis, done by the statistician. (Hence p-values but not alpha and power analysis). However, the decisions about how to behave, whether to carry on with research etc. are made by the scientist on the basis of her understanding of domain theory, not by the statistician applying the inference paradigm. Because of this Fisherian division of labour, both the subjectiveness (sense 2) and the individual subject (sense 1) sit on the science side, not the statistical side.

Legalistically speaking, the Fisher's Frequentism is subjective. It's just that the subject who is subjective is not the statistician.

There are various syntheses of these available, both the barely coherent mix of these two you find in applied statistics textbooks and more nuanced versions, e.g. the 'Error Statistics' pushed by Deborah Mayo. This latter is pretty unsubjective in sense 2, but highly subjective in sense 1, because the researcher has to use scientific judgement - Fisher style - to figure out what error probabilities matter and shoudl be tested.

Frequentism as subjective (first sense)

So is Frequentism less subjective in the first sense? It depends. Any inference procedure can be riddled with idiosyncracies as actually applied. So perhaps it's more useful to ask whether Frequentism encourages a less subjective (first sense) approach? I doubt it - I think the self conscious application of subjective (second sense) methods leads to less subjective (first sense) outcomes, but it can be argued either way.

Assume for a moment that subjectiveness (first sense) sneaks into an analysis via 'choices'. Bayesianism does seem to involve more 'choices'. In the simplest case the choices tally up as: one set of potentially idiosyncratic assumptions for the Frequentist (the Likelihood function or equivalent) and two sets for the Bayesian (the Likelihood and a prior over the unknowns).

However, Bayesians know they're being subjective (in the second sense) about all these choices so they are liable to be more self conscious about the implications which should lead to less subjectiveness (in the first sense).

In contrast, if one looks up a test in a big book of tests, then one could get the feeling that the result is less subjective (first sense), but arguably that's a result of substituting some other subject's understanding of the problem for one's own. It's not clear that one has gotten less subjective this way, but it might feel that way. I think most would agree that that's unhelpful.

-
A dictionary definition of 'subjective' (straight from Google) says: based on or influenced by personal feelings, tastes, or opinions. e.g. "his views are highly subjective" synonyms: personal, personalized, individual, internal, emotional, instinctive, intuitive, impressionistic. Notice that this reflects a folk theory that a 'view' (ie a belief) is subjective (sense 1: impressionistic, intuitive, weird etc.) because it concerns the internal state of a particular subject (sense 2: personalized, individual, etc.) rather than being public ie impersonal. – conjugateprior Jul 15 '14 at 13:14
It might be helpful to think about cognitive psychology as an example. This field is completely subjective (in second sense, because it's all about the internal states of people and their effects on those people's behaviour) but it's not subjective in the first sense, because psychologists can't actually just sit around and make stuff up on the basis of their own internal state. – conjugateprior Jul 15 '14 at 13:20
The opposite extreme, where something is totally idiosyncratic and subjective (sense 1) but not actually about subjects at all is trickier to find. Perhaps Lucretius explaining atoms and void in de Rerum Naturae is an example. – conjugateprior Jul 15 '14 at 13:26
and I've approved (and adjusted slightly) the proposed alteration – conjugateprior Jul 15 '14 at 15:31

The subjectivity in frequentist approaches is rampant in application of inference. When you test a hypothesis you set a confidence level, say 95% or 99%. Where does this come from? It doesn't come from anywhere but your own preferences or a prevailing practice in your field.

Bayesian prior matter very little on large datasets, because when you update it with the data, the posterior distribution will float away from your prior as more and more data is processed.

Having said that Bayesians start from subjective definition of probabilities, beliefs etc. This makes them different from frequentists, who think in terms of objective probabilities. In small data sets this makes a difference

UPDATE: I hope you hate philosophy as much as I do, but they have some interesting thought from time to time, consider subjectivism. How do I know that I'm really on SE? What if it's my dream? etc. :)

-
Aside from the choice of a confidence level in hypothesis testing (since the same could be argued in Bayesian statistics, for example, when choosing a criteria for comparing HDP/HDI with ROPE for rejecting or accepting a hypothesis), does subjectivity play a role in getting a point estimate or obtaining confidence intervals, perhaps in the choice of estimators? – Amelio Vazquez-Reina Jul 14 '14 at 0:23
Also, my understanding is that significance levels are set in Frequentist statistics in the context of decision making (i.e. should we reject the null hypothesis?), not in computing probabilities. In Bayesian decision theory the same could be argued about the choice of a Loss function, which can affect the optimal (chosen) decision. Moreover, confidence level values are usually chosen from an acceptable type I error rate (e.g. 95% in NHST is directly established from a "no higher than 5%" false positive rate) – Amelio Vazquez-Reina Jul 14 '14 at 0:29
Certainly the prior matters quite a bit when analyzing large datasets, for most interesting modern problems. True, we have asymptotic results when $n \to \infty$ and $p \ll n$, but typically people with big datasets ask big questions so that in some sense $p \to \infty$ at a rate comparable to $n$, and here an appropriate prior is going to act like a regularizer and will matter a great deal. – guy Jul 14 '14 at 0:31
It might be worth noting that loss functions may not be subjective (i.e., completely determined by the context), in which case Bayesian optimal decisions have the potential to be completely objective apart from the prior. – Matt Jul 14 '14 at 0:36
@Matthew Yes, although that also goes true for the choice of $\alpha$ in NHST from the maximum tolerable false positive rate, as I mentioned above. – Amelio Vazquez-Reina Jul 14 '14 at 0:40