1
$\begingroup$

Suppose you use Bayesian methods to calculate the probability density function (pdf) of a quantity of interest $X$, given its measured value $x$ (measured only once) and some other assumptions/constraints/knowledge about the experiment.

Can you then use that pdf to calculate an expected value for $X_{exp}$ (integral mean of pdf(X) * X over the whole domain) and some interval, taking e.g. $X_{left} < X_{exp} < X_{right}$ that result in a given % of the integral of the pdf?

Why the doubt? Because the people who run the experiment argue that:

  1. it does not make any sense to report uncertainty for $n = 1$
  2. the variability on repeats of the experiment is often much larger than the one resulting from the Bayesian pdf, so one would incur the (in their opinion absurd) scenario where uncertainty could increase as more experiments are run.
    (Note: it is true that the biological material used in the assay can behave differently depending on how it was sourced, handled, etc., so if differently behaving material is used in different runs of the experiment, the results can indeed vary a lot).

My counterargument is that we are not making any false statements if we use the pdf-derived interval for $n = 1$: we are just saying that, based on the evidence collected in this single experiment, the 'true' value $X$ for this specific batch of biological material is believed to be between such and such bounds.
If next time we run the experiment the biological material behaves very differently, resulting in a very different $X_{exp}$, IMO that is no falsification of the earlier statement, but just a manifestation of the actual non-homogeneity of the two batches of material.
It is only because we decide to treat different runs of the experiment with different biological material as measurements of the 'same' quantity, i.e. we average all $X_{exp}$'s, that we run the risk to have increased uncertainty for $n>1$ than we had for $n=1$.
This apparent contradiction is only a consequence of our own choice to aggregate data from different and maybe incompatible experiments; or am I wrong?

Besides, there is nothing stopping uncertainty from increasing on repeats of the experiments even when the measured $x$ is used, not the Bayesian-derived $X$.
So in fact I would argue that using a Bayesian method, by producing a pdf, enables one to report some rationally derived estimate of uncertainty for the (much more frequent BTW) $n = 1$ cases, whereas sticking to the measurement $x$ only allows reporting uncertainty when $n > 1$, and worse, one is then hostage of the specific batches of material that were used in the experiments, potentially resulting in far over- or under-estimated confidence intervals.

I would really appreciate to hear the view of this expert community on this topic, which is causing quite some discussion and controversy in our company.

$\endgroup$

1 Answer 1

2
$\begingroup$

You won't be able to calculate a (frequentist) confidence interval from a single observation since you cannot calculate two unknowns (mean $\bar{x}$ and variance $s^{2}$ needed for the normal approximation $\bar{x} \pm z^\ast \left(s / \sqrt{n} \right) $) with just one observation.

However, in Bayesian inference you're starting with a prior probability which is already a "full" probability distribution. Hence, to paraphrase Richard McElreath, the minimum sample size required for Bayesian inference is $n=0$.

Both with the prior $p(X)$ as well as with the posterior $p(X|x_{1})$ (where $x_{1}$ is your single data point) you will be able to calculate $X_{exp} = \int X p(X)d{X}$ or $X_{exp'} = \int X p(X|x_{1})d{X}$, respectively, or any credible interval. They will obviously be heavily influenced by your prior, but it is possible to calculate these quantities.

$\endgroup$
4
  • 3
    $\begingroup$ Regarding confidence intervals for $n=1$ or $n=2$, see this paper. I don't know if that is relevant to the question but I decided to link to it after reading your first sentence. $\endgroup$ Commented Feb 15 at 18:32
  • $\begingroup$ Great find! Indeed, I should've been more precise in that opening sentence: "you cannot calculate the standard confidence interval from a single sample without further (distributional) assumptions", which is what your linked paper describes. In lieu of a sufficiently large sample, the missing information has to come from extraneous assumptions, whether those are called priors or something else. $\endgroup$
    – Durden
    Commented Feb 15 at 19:45
  • $\begingroup$ @COOLSerdash : excellent, thanks, I requested the paper via our internal system, I will see what they say about this. $\endgroup$ Commented Feb 16 at 10:59
  • $\begingroup$ @Durden : thanks, I know I can calculate it, the problem is that people are not OK with reporting it. But perhaps the paper mentioned here will set the record straight. In fact, even before using Bayesian methods (which in this case are needed also for other reasons that are long to explain), I had considered using some information about the 'historical' variability of some assays to make an estimate of the confint for n = 1. That amounts to about the same, in practice, when the measurement falls far from the bounds. But that was rejected as well by the people who run the assay. $\endgroup$ Commented Feb 16 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.