6
$\begingroup$

We are usually told the following:

  • In the Frequentist Probability Approach, we are told that: the data is random but the parameters being estimated are fixed
  • In Bayesian Probability Approach, we are told that: the data is fixed but the parameters being estimated are random

Conceptually, both approaches have their advantages:

  • Often in the real world, the data we collect can be thought of as random - for example, if we collect the same data on a different day, the data might not be exactly the same as the data from another day. Therefore, treating the data as random might have its benefits.

  • On the other hand, the parameters we are trying to estimate can also be thought of as random - many times the "true" values of these parameters are not directly observable and might inherently have some level of randomness encoded within themselves. Therefore, treating the parameters as random might also have its benefits.

I was wondering about the following question: Is it possible to combine both of these approaches together such that both the data and the parameters are considered as random?

This way we can get the best out of both worlds?

Thanks!

$\endgroup$
5
  • 3
    $\begingroup$ Really it would be the worst of both worlds, because if both the parameter and the data are random quantities, to define an estimand it would have compounded variability. For instance, a simple quantity like $Var(\theta)$ becomes $var(E(\theta|X)) + E(var(\theta | X))$. $\endgroup$
    – AdamO
    Commented May 3, 2023 at 5:05
  • 10
    $\begingroup$ In both the frequentist and Bayesian approaches that data is a realization of a random variable. There is no difference between them in that regard., In the Bayesian approach parameters are also considered to be random variables. $\endgroup$
    – J. Delaney
    Commented May 3, 2023 at 10:42
  • 1
    $\begingroup$ @J.Delany While your comment is technically correct in regards to observed data, it downplays the differences between the two approaches. Perhaps a better shorthand for the Frequentist approach is: "Frequentist applications often assume data is generated as if by a process involving randomness and one or more unknown parameters." As we know, random variables need not involve randomness. $\endgroup$ Commented May 4, 2023 at 11:57
  • 1
    $\begingroup$ @GrahamBornholt, in which sense do you mean random variables need not involve randomness? As in epistemic uncertainty that allows using probabilities to characterize nonrandom objects (OK among Bayesians, not OK among frequentists), contrary to aleatory uncertainty which uses probabilities only to characterize random objects (OK among both Bayesians and frequentists)? $\endgroup$ Commented May 9, 2023 at 7:17
  • $\begingroup$ I was simply making the point that according to its definition, the term "random variable" does not imply randomness was/is involved. In contrast, for example, the term "random sample" does. Perhaps I am mistaken $\endgroup$ Commented May 12, 2023 at 9:46

2 Answers 2

6
$\begingroup$

You can look at the a priori inferential properties of estimators (which treats both the data and parameters as random), but this is weaker than standard analysis

If you look at a statistical problem from a perspective where both the data and the model parameters are treated as random, you are essentially looking at the a priori properties of estimators. This is an exercise that can be done fruitfully, and it falls within the general class of analysis of the Bayesian properties of estimators. However, performing analysis of this kind is typically weaker than looking at the classical properties of estimators.

To see what this type of analysis looks like, suppose we consider some estimation/inferential method, which as you point out, is built on the basis of its properties conditional on the model parameters but unconditional on the data. For example, an exact confidence interval for a model parameter $\theta \in \Theta$ (based on a data vector $\mathbf{x}$) would have the following property (which is essentially the defining property of an exact confidence interval):

$$\mathbb{P}(\theta \in \text{CI}( \mathbf{X}, \alpha) | \theta ) = 1-\alpha \quad \quad \quad \text{for all } \theta \in \Theta.$$

Now, if we take any prior distribution $\pi$ for the model parameter then the above property implies the weaker property:

$$\mathbb{P}(\theta \in \text{CI}( \mathbf{X}, \alpha)) = \int \limits_\Theta \mathbb{P}(\theta \in \text{CI}( \mathbf{X}, \alpha) | \theta ) \cdot \pi(\theta) \ d\theta = 1-\alpha.$$

As you can see, because the coverage property for a CI holds under all specific parameter values $\theta$ (which is how we analyse estimators/inference methods in classical analysis), this implies that it must also hold (marginally) for any prior distribution over the possible values of $\theta$. Note that the latter is a weaker property than the underlying property defining the exact confidence interval, but it is interesting to note. This tells us that an exact confidence interval formed by classical methods is such that a priori we expect it to have the correct coverage. This is what it looks like to analyse the properties of a statistical estimator treating both the data and the parameter as random.

I note your overarching question about whether it would be possible to form a new hybrid approach to estimation/inference by combining classical methods and Bayesian methods. That might be possible in principle, but because the above a priori analysis is weaker than the standard classical approach to looking at estimation, it is unlikely that this would assist you to formulate a better method than existing approaches.

$\endgroup$
5
$\begingroup$

As J. Delaney's comment says, the Bayesian approach already allows both the data and the parameters to be random.

I think the confusion arises because "the parameters are fixed and the data is random" is not true under the frequentist approach, and "the parameters are random and the data is fixed" is not true under the Bayesian approach either. (See the answers to this question for more details.)

What is going on? In both cases you choose a family of models, for example a $N(\mu, \sigma^2)$, which could have generated your data $X$.

In the Bayesian case, you treat $\mu$ and $\sigma^2$ as random variables and calculate their conditional distribution given your observed data $X$. In order to do this, you must choose a prior distribution for $\mu$ and $\sigma^2$. Sometimes you don't want to do this.

In the Frequentist case, you are not allowed to treat $\mu$ and $\sigma^2$ as random variables. Instead, you seek to make statements which are valid no matter what the true values of $\mu$ and $\sigma^2$ happen to be. These statements are constructed by considering what kind of data might have been generated by different values of $\mu$ and $\sigma^2$. But whatever result you get is still conditional on your observed data $X$. It's just that it's not called a conditional distribution in the frequentist case.

For example, suppose your frequentist confidence interval for $\mu$ is $[2, 3]$. Then if you had collected a different data set $X'$ on a different day, you would probably end up with a different confidence interval. Similarly, say your Bayesian credible interval for $\mu$ is $[2, 3]$. If you had collected a different data set $X'$ on a different day, you would probably end up with a different credible interval as well.

$\endgroup$
1
  • 1
    $\begingroup$ A realization of a random variable is not random. So you have misunderstood J. Delany's comment $\endgroup$ Commented May 4, 2023 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.