11
$\begingroup$

In my opinion, in BOTH Bayesian and Frequentist inferences, observational data $x$ are modelled as the observed value of a random variable $X$ which follows a certain probability distribution. Therefore, when I encounter the two statements - "Data are fixed" in the Bayesian viewpoint, and "Data are random" in the frequentist viewpoint, I found them challenging to understand since it appears to suggest a discrepancy in the interpretation of the concept of data.

After reading some materials, it seems that “data are fixed” means the observed data $x$ is fixed and “data are random” means the random variable $X$ is random. The “data” in the two statements are not referring to the same thing. Not sure if my interpretation is correct. Advice from you is welcome. If my interpretation is correct, is it redundant/confusing to make these two statements?

Edits

Reference of the two statements:-

Improve this question
$\endgroup$
8
  • 5
    $\begingroup$ Comparing the meaning of a frequentist $95\%$ confidence interval and Bayesian $95\%$ credible interval might help understand the simplification inherent in your quotations. $\endgroup$
    – Henry
    Commented Apr 7 at 12:27
  • 1
    $\begingroup$ In Bayesian statistics, it doesn't really matter if you consider the data as random or as fixed. What matters is that you condition on the observed values, which is mathematically equivalent to saying they're constant. On the other hand, a frequentist decision must be right 95% of the time when the experiment is repeated. In order to derive or validate a decision rule, you have to consider the chances when the experiment is repeated: for this, you have to consider the data as random variables. Only when the general rule is established you can apply it to the observed values. $\endgroup$
    – jacques
    Commented Apr 8 at 15:42
  • 1
    $\begingroup$ My professor gave this example of a problem which, in his opinion, is only amenable to a Bayesian treatment: It's not known in which order all Plato's 6 major works are published. But one or two are known, and there are clear shifts in his writing style. So you can use all the usual trend estimators to suggest a ranking, but quantifying your belief in the ranking requires Bayesian thought only. A frequentist can't really talk about an infinite number of Platos, with 6 publications, and no observable trend in writing style. $\endgroup$
    – AdamO
    Commented Apr 8 at 19:53
  • 1
    $\begingroup$ @AdamO One can talk about a hypothetical population of Platos. It is very much like the Socratic method. You start with a hypothetical believe and then falsify it based on dialogues that show inconsistencies (or in the case of statistics based on data). Also, a Bayesian method uses just as well the idea of a distribution for the data like a frequentist method does. Otherwise the method doesn't have a likelihood function to compute the posterior. The likelihood function is a function of probabilities, and that's just as well contemplating an infinite number of Platos. $\endgroup$
    – Sextus Empiricus
    Commented Apr 8 at 20:11
  • $\begingroup$ @AdamO: I would also argue that Bayesian statistics is only rational when it also follows a Frequentist interpretation: if you consider the set of events that you give probability 0.9 of being true, and on average a very different proportion are true...do I really want your prior impacting my inference? Why would I ever care about your posterior belief? You just need to pivot of the set you are considering a measure over and you could can make the Frequentist interpretation a requirement for rational belief in a Bayesian viewpoint. $\endgroup$
    – Cliff AB
    Commented Apr 9 at 5:16

7 Answers 7

Reset to default
15
$\begingroup$

The usage of the expressions 'data are fixed' or 'parameters are fixed' in the linked references should not be taken literally.

  • Data is just as well considered random in Bayesian analyses, how else would the use of the likelihood in Bayes theorem make sense?
  • Parameters may just as well be considered as a random variable in frequentist analyses. (for instance, priors could be used to minimize the expectation value of the length of confidence intervals)

A difference is that they consider different marginal distributions of the joint distribution of data and parameters when computing intervals. It is not frequentist versus Bayesian, but the credible interval versus the fiducial/confidence interval.

You can see an example of this difference in this plot from the question Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals

Wasserman example

The image considers a joint distribution of parameters $\theta$ and data $X$.

  • A credible interval will be computed by considering the distribution of $\theta$ given $X$ and considers horizontal slices of the joint distribution. For each value $X$ the boundaries contain 95% of the potential values $\theta$.

  • A confidence interval will be computed by considering the distribution of $X$ given $\theta$ and considers vertical slices of the joint distribution. For each value $\theta$ the boundaries contain 95% of the potential values $X$

It is this conditioning that relates to the parameters or data being described as 'fixed'. The Bayesian credible interval considers a distribution of $\theta$ given a fixed value of $X$. The frequentist confidence interval considers a distribution of $X$ given a fixed value of $\theta$ (and actually, the interval, considers many of such fixed values, the interval is the collection of values $\theta$ for which the sample distribution of the data $X$ has certain properties that align with the observed data).

As a consequence of this conditioning the coverage probability of the intervals will be different depending on whether we consider the coverage if the data is a certain value $X$ or the coverage if the parameter is a certain value $\theta$.

Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

Reviewing some of my old answers I notice that I have used the expression 'fixed' myself as well. It is in an answer to this question: If a credible interval has a flat prior, is a 95% confidence interval equal to a 95% credible interval?

In that answer I made use of the image below to describe a difference between the construction of credible and confidence intervals for the example case of an exponential distribution of the data and a uniform prior for the parameter.

It is not like the data or parameters are neccesarily considered to be truly fixed, but it is just that the intervals relate to computations of conditional distributions where the one (data/parameter) is computed conditional on the other (parameter/data) being fixed.

difference between credible and confidence intervals

Why do all this trouble for confidence intervals instead of using credible intervals? What is the point about the computations with the parameter fixed?

The advantage of frequentist confidence intervals, because they rely on computations conditional on the parameter, can be independent of assumptions about a distribution of that parameter.

(Although there can be philosophical differences in ideas about probability, it is not neccesarily the case that a statistician using a frequentist confidence interval believes that the parameter is fixed. It is more that the statistician doesn't need to make use of any assumptions about the distribution of the parameter.)

Improve this answer
$\endgroup$
4
  • $\begingroup$ I guess I got the idea. Confidence interval in frequentist is derived from $P(X | \theta)$ (or vertical lines) so we say "parameter is fixed; data is random". Credible interval in bayesian is derived from $\pi(\theta |x)$ (or a horizontal line), so we say "parameter is random, data is fixed". AND if I want to get the confidence interval after obtaining an observation $x$, I should actually draw a horizontal line and find the corresponding range of $\theta$ bound by the two red lines in the 1st plot. Am I correct? Moreover, I don't quite understand your 2nd plot. Can u pls explain? $\endgroup$
    – Ken T
    Commented Apr 9 at 18:07
  • 1
    $\begingroup$ @KenT „Confidence intervals are derived from P(X|θ). Credible intervals are derived from P(θ|x)“ That is a nice succinct description. (and while they are derived with those different expressions it doesn't necessarily mean that they regard X or θ as intrinsicly fixed, it is only in the used expressions that the parameter is fixed.) $\endgroup$
    – Sextus Empiricus
    Commented Apr 9 at 19:10
  • 1
    $\begingroup$ I will see if I can explain the second graph better. For the moment, say that an observation is $\bar{x} = 1$ then the intervals are based on the value of $\theta$ where the boundary lines cross the value $\bar{x} = 1$ . See another illustration here: Confidence Intervals: how to formally deal with $P(L(\textbf{X}) \leq \theta, U(\textbf{X})\geq\theta) = 1-\alpha$ $\endgroup$
    – Sextus Empiricus
    Commented Apr 9 at 19:18
  • $\begingroup$ Another occurrence of this type of graph is here: stats.stackexchange.com/a/351330 It is noteworthy because of the final paragraphs that mention: „In the vertical direction you see hypothesis testing” and „In the horizontal direction you see Clopper-Pearson confidence intervals.“. The confidence intervals consider the conditional probabilities P(X|θ) with theta fixed, but at the same time they consider this at a range of different values of theta and theta may potentially be considered as a variable. $\endgroup$
    – Sextus Empiricus
    Commented Apr 9 at 19:38
11
$\begingroup$

No they are not talking about the same thing. This is readily seen in sequential experimentation where frequentist statistics takes an $\alpha$ (type I assertion probability) penalty for multiple data looks, because multiple looks give more opportunities for data to be extreme (p-values to be low). With Bayes, data are conditioned on, and at each look you compute the probability of a positive effect conditional on current cumulative data. The meaning of these posterior probabilities doesn’t change over the course of the multiple looks, and past data are merely deemed obsolete, with current cumulative data overriding whatever you knew at an earlier look.

Improve this answer
$\endgroup$
5
  • 6
    $\begingroup$ Why wouldn't the Bayesian analog of "multiple looks" be multiple hypotheses, expressed (perhaps) in the forms of varying priors and varying models? Ineluctably, if one snoops using Bayesian methods one can arrive at facially convincing posteriors that reflect only the persistence of the investigator in attempting to achieve a desired result! $\endgroup$
    – whuber ♦
    Commented Apr 7 at 15:04
  • $\begingroup$ No, it's multiple repeated analyses of the same thing. So there is only one prior, and sampling to a forgone conclusion only happens if the judge has a prior with an absorbing state (a spike in the prior) that the analyst doesn't use. Details here. Bayesian sequential analysis just represents updates that treat previous analyses obsolete. $\endgroup$
    – Frank Harrell
    Commented Apr 8 at 16:07
  • 1
    $\begingroup$ Thank you for responding. I think I am struggling with what you mean by each "look." I think you might not be making a fair comparison: there are frequentist methods (originating with Wald) for sequential experimentation that rigorously control $\alpha$ for the entire experiment. Thus it seems you are contrasting an invalid application of frequentist procedures to a valid application of Bayes procedures. // The use of "deemed obsolete" is particularly puzzling: after all, wouldn't the final Bayes result be the same under any permutation of the data? $\endgroup$
    – whuber ♦
    Commented Apr 8 at 16:13
  • 2
    $\begingroup$ Wald's sequential likelihood ratio test is a powerful approach but it doesn't apply to multiple outcome variables and can't let you use prior information from other studies. More to the point, it has never been accepted in the clinical trials world where group sequential testing is the norm, and super emphasis is placed on $\alpha$-spending. This involves a penalty for each look, something foreign to Bayes. Yes in the absence of a modeled time trend Bayes is data-order-independent but the posterior probability computed earlier is obsolete and ignored. A reason for no penalty. $\endgroup$
    – Frank Harrell
    Commented Apr 9 at 12:42
  • 1
    $\begingroup$ Thank you for the explanation and clarification -- much appreciated. $\endgroup$
    – whuber ♦
    Commented Apr 9 at 13:01
11
$\begingroup$

I suppose what they mean (it may help to cite your sources) is that Bayesian inferences involve posterior probabilities conditioned on the observed data. Frequentist inferences are typically conditional on only part of the observed data, if any. Note though that in randomization tests the data are considered fixed, while the assignment to experimental treatments is random.

I'm not sure it's especially helpful to say "data are fixed" w.r.t. Bayesian inference. As you point out, there's still a probability model for the data-generating process that needs to be stipulated; unlike the way in which other things are said to be fixed: the predictors in a regression model, say, where it's immaterial whether their values arise from sampling or are set by the experimenter. As for "data are random" w.r.t. frequentist inference, careful discussion distinguishes between data as potential observations (random variable) & data as actual observations (its realization)—consider e.g. estimator vs estimate.

It perhaps makes most sense in contrasting the probabilities &c. involved in interpreting the results of Bayesian vs frequentist analyses, as @Henry points out. What's fixed & what's random in a confidence interval? In an H.P.D. interval? Durden's answer explains nicely.

Improve this answer
$\endgroup$
11
$\begingroup$

After reading some materials, it seems that “data are fixed” means the observed data 𝑥 is fixed and “data are random” means the random variable 𝑋 is random. The “data” in the two statements are not referring to the same thing.

This isn't quite right, and the phrase is meant to be a little more tongue-in-cheek than literal.

Why is the data random in the Frequentist perspective?

Both Frequentist and Bayesians agree that their models assume that the data is generated from some random process, such as $X \sim N(\mu, \sigma)$. In the Frequentist world, this is the only source of randomness, since it is considered that $\mu$, $\sigma$ are just unknown but fixed values. So statistics deal with making sense of randomness, and the only randomness in a Frequentist perspective is from $X$.

Why is data fixed in the Bayesian perspective?

Where the Bayesian perspective differs is that it is willing to consider $\mu$, $\sigma$ as random variables as well. And while $X$ was considered a random variable before the value was known, it is now observed so it is no longer random: the outcome from a coin flip is random, but the outcome from a coin flip that landed on heads is not. So once you have observed your data, the only randomness that remains is in the uncertainty of the parameters of interest.

A more clear way to phrase this would be "Both Frequentist and Bayesian perspective agree that the data was random before it was known, but only Bayesian consider that the value of the parameters of interest are still random." Doesn't sound quite as cute though.

Improve this answer
$\endgroup$
1
  • $\begingroup$ "Where the Bayesian perspective differs is that it is willing to consider 𝜇, 𝜎 as random variables as well." This is correct, but I would add a clarification: Frequentists understand probability as a limit of an observable frequency of outcomes. On the other hand, Bayesians understand probability more broadly as a way to encode subjective (w.r.t. available information) uncertainty. A Bayesian does inference based on available information on the parameter value, whereas a Frequentist does inference based on a general algorithm that is guaranteed to be right 95% of the time. $\endgroup$
    – jacques
    Commented Apr 8 at 15:38
6
$\begingroup$

In both Bayesian and frequentist stats, your current sample $x$ is fixed while potential other samples (i.e., future realizations of $X$) are random.

Frequentists rely on the idea of resampling for the interpretation of the probabilistic properties of estimators and test statistics. Take confidence intervals for example: they are calculated from the sample, so their randomness derives from the randomness of sampling from $X$. For a given sample $X=x$, on the other hand, they are fixed; either containing the true parameter or not (but you have no way of knowing which). Without the idea of the data being random, the probability statement of a "95% confidence interval" would have no meaning.

Bayesians, on the other hand, don't rely on this construct. The probability statement of a credible interval refers to your knowledge after taking the actually existing sample into account. Potential resampling of $X$ plays no role and does not affect your current state of knowledge.

Improve this answer
$\endgroup$
13
  • 2
    $\begingroup$ (+1) Nicely put. Could say the same, mutatis mutandis, about estimators & tests. $\endgroup$
    – Scortchi ♦
    Commented Apr 7 at 15:59
  • 2
    $\begingroup$ 'the probability statement "95%" refers to hypothetical confidence intervals calculated with those future samples (even if impossible and thus imaginary)' is a common misconception. The "95%" is a pre-sample implication of the model about possible samples but no resampling is required for its validity or interpretation. Afterall, we can consider the performance of the CI method in a large number of applications of the CI method across independent problems. $\endgroup$
    – Graham Bornholt
    Commented Apr 7 at 19:43
  • 2
    $\begingroup$ .... Once our sample is collected, there is no more randomness so no more probability, post-sampling. We know our calculated CI either does or does not cover the parameter but the fact that it is the outcome of a procedure that gets it right 95% of the time (given the model(s)) provides some confidence. Now “95% of the time” could refer to hypothetical resampling or to actual reuse of the procedure across different problems, but either way whether or not resampling is possible is of no consequence because resampling is not a necessary component of the CI method. $\endgroup$
    – Graham Bornholt
    Commented Apr 8 at 20:46
  • 2
    $\begingroup$ The discussion on the resampling in frequentist is very interesting. I think resampling is not necessary in frequentist method. Let say we design an experiment of measuring the weight of 10 birds out of 1,000 and will release all 1,000 birds afterwards. In this experiment, resampling is not possible but we will still assume that the weight of a bird is an observed value of a random variable which has a certain probability distribution. As long as it is still a random variable, all statistical inference of frequentist is still applicable. $\endgroup$
    – Ken T
    Commented Apr 9 at 10:13
  • 2
    $\begingroup$ It appears then we've reached common ground. And to tie it back to @KenT's original question as well as my answer above: the assumption that $x$ is but one realization of $X$ is fundamental to (the interpretation of) frequentist statistics, while being immaterial for Bayesians. $\endgroup$
    – Durden
    Commented Apr 9 at 19:52
6
$\begingroup$

OP says:

“data are fixed” means the observed data $x$ is fixed and “data are random” means the random variable $X$ is random. The “data” in the two statements are not referring to the same thing.

That's a perfectly reasonable way to think about it.

  • Bayesian tools are meant to summarize uncertainty about the parameter $\theta$, conditional on the observed data $x$.
  • Frequentist tools are meant to report properties of the study design that can be thought of as properties of the random variable $X$... or more typically, properties of the estimator $\hat\theta$, which is itself treated as a random variable that is a function of $X$.

Frequentist summaries of uncertainty/precision are, essentially, summaries of the sampling error associated with the design of your study (and the population being studied). They focus on the "aleatory uncertainty" alone.

I find this can be clearer if we talk about the margin of error in its own right, rather than the confidence interval. A physical measurement instrument gets tested in a lab before widespread use, and is sold with a published estimate of precision: "This kitchen scale is accurate to within 0.1 grams!" Similarly, a statistical study design can be carefully chosen to have a particular margin of error: "Sample means from this study design should be within 0.3 units of the true population mean 95% of the time!"

The margin of error depends on various aspects of study design, such as the sample size, or the use of stratification (in survey sampling) or blocking (in designed experiments), and so on... but in many simple cases, the Frequentist margin of error does not depend on the parameter value. So it's often fine to treat the parameter as some fixed (but unknown) value as you work out the math to calculate the margin of error. Hence the Frequentist's shorthand phrase:

  • "The data are random..." (you've deliberately chosen to focus on modeling the data given a parameter value in terms of aleatory randomness in the data)
  • "...but the parameter is fixed." (your margins of error, confidence intervals, or other uncertainty measures should work conditionally on the parameter value no matter which value it is)

On the other hand, Bayesian summaries of uncertainty/precision typically account for the "epistemic uncertainty" as well. What did I believe about the parameter before I saw this new dataset, and how should I update my belief conditional on this data?

The prior and posterior are both summaries of your state of knowledge of the parameter. Hence the Bayesian's shorthand phrase:

  • "The parameter is random..." (it's not necessarily random in the aleatory sense, but in the epistemic sense -- you are deliberately choosing to use a distribution to summarize your knowledge/beliefs about the parameter)
  • "...but the data are fixed" (not literally -- you often also have an aleatory and/or epistemic model for the data -- but you are focused on reporting a posterior distribution which is conditional on the data you just saw)

Now, by focusing only on aleatory uncertainty, Frequentist tools omit other legitimate sources of uncertainty. But if you separate out the different sources of uncertainty, you can check each of them individually, making sure there are no weak links in the chain of evidence.

If instead you only provide Bayesian summaries of uncertainty, you might have a harder time assessing the aleatory-uncertainty link in particular, because Bayesian tools combine both the epistemic and aleatory uncertainty into one joint summary.

Improve this answer
$\endgroup$
-1
$\begingroup$

What are data? Data come in all shapes and sizes. But what are data actually?

My height and weight can be measured, and their values are read using a scale and a ruler. The values can be recorded with a pencil on a sheet of paper or typed on my computer. In any case, the readings are only registered with finite precision. This is what we call data, namely finite-precision numbers.

Data are an ontological concept. Data are part of the real world. Therefore, data are fixed.

Probability is an epistemological concept. Epistemology is the study of the nature of knowledge, justification, and the rationality of belief.

"Randomness" is in our minds and is not part of the physical world. Random variables are therefore a fundamentally different concept than data.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ Thanks for your advice. I am not asking for the general definition of data but the particular meaning of the “data” in the two statements. But what you have mentioned about data is interesting. In my opinion, data has different layers of meaning. Some may refer data to the intrinsic measurable property of an entity, like height of a person. I believe data in statistics mean the observed values from an experiment. Like an experiment to pick and measure heigh of 10 people out of 1000. While the height of a person is not random, the picking process creates random. $\endgroup$
    – Ken T
    Commented Apr 7 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.