In Bayesian data analysis, parameters are treated as random variables. This stems from the Bayesian subjective conceptualization of probability. But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'
It seems like the obvious answer is 'yes', because then trying to estimate the parameter would almost be nonsensical. An academic citation for this answer would be greatly appreciated.
IMHO "yes"! Here is one of my favorite quotes by Greenland (2006: 767):
It is often said (incorrectly) that ‘parameters are treated as fixed by the frequentist but as random by the Bayesian’. For frequentists and Bayesians alike, the value of a parameter may have been fixed from the start or may have been generated from a physically random mechanism. In either case, both suppose it has taken on some fixed value that we would like to know. The Bayesian uses formal probability models to express personal uncertainty about that value. The ‘randomness’ in these models represents personal uncertainty about the parameter’s value; it is not a property of the parameter (although we should hope it accurately reflects properties of the mechanisms that produced the parameter).
Greenland, S. (2006). Bayesian perspectives for epidemiological research: I. Foundations and basic methods. International Journal of Epidemiology, 35(3), 765–774.
The Bayesian conception of a probability is not necessarily subjective (c.f. Jaynes). The important distinction here is that the Bayesian attempts to determine his/her state of knowledge regarding the value of the parameter by combining a prior distribution for its plausible value with the likelihood which summarises the information contained in some observations. Hence, as a Bayesian, I'd say that I am happy with the idea that the parameter has a true value, which is not known exactly, and the purpose of a posterior distribution is to summarise what I do know about its plausible values, based on my prior assumptions and the observations.
Now, when I make a model, the model is not reality. So in some cases the parameter in question does exist in reality (e.g. the average weight of a wombat) and in some questions it doesn't (e.g. the true value of a regression parameter - the regression model is only a model of the outcome of the physical laws that govern the system, which may not actually be captured fully by the regression model). So to say that there is one true fixed parameter value in the real world is not necessarily true.
On the flip side, I would suggest that most frequentists would say there is one true value for the statistic, but they don't know what it is either, but they have estimators for it and confidence intervals on their estimates which (in a sense) quantifies their uncertainty regarding the plausibility of different values (but the frequentist conception of a probability prevent them from expressing this as directly).
To your main point, in Bayesian Data Analysis (3rd ed., 93), Gelman also writes
From the perspective of Bayesian data analysis, we can often interpret classical point estimates as exact or approximate posterior summaries based on some implicit full probability model. In the limit of large sample size, in fact, we can use asymptotic theory to construct a theoretical Bayesian justification for classical maximum likelihood inference.
So perhaps it's not Bayesians who should "admit" that there are, in truth, single real parameter values, but frequentists who should appeal to Bayesian statistics to justify their estimation procedures! (I say this with tongue firmly in cheek.)
As an aside, I object to the blanket statement that Bayesian statistics is premised on subjective probability, and implication that Bayes is subjective while other inferential paradigms are not. That is certainly one argument that can be posed, perhaps also including the perspective of the "coherence of bets" argument, but see Gelman who here defines "Bayesian" as a statistician that uses the posterior distribution $\Pr(\theta|y)$, and here where he argues against overly restrictive definitions.
But the idea that there are single parameters in nature or in social systems is just a simplifying assumption. There might be some ornate process generating observable results, but discovering that system is incredibly complicated; supposing that there is a single fixed parameter value simplifies the problem dramatically. I think that this cuts to the core of your question: Bayesians shouldn't have to "admit" to making this simplification any more than Frequentists should.
Do you think that there is a single "true fixed parameter" for something like the contribution of milk drinking to a child's growth? Or for the decrease in a tumor's size based on the amount of chemical X you inject into a patient's body? Pick any model you're familiar with and ask yourself if you actually believe that there is one true, universal, precise and fixed value for each parameter, even in theory.
Ignore measurement error, just look at your model as if all measurements were perfectly accurate and infinitely precise. Given your model, do you think that each parameter realistically has a specific point value?
The fact that you have a model indicates that you are leaving some details out. Your model will have an amount of imprecision because you're averaging over the parameters/variables that you've left out in order to make a model -- a simplified representation of reality. (Just as you don't make a 1:1 map of the planet, complete with all details, but rather a 1:10000000 map, or some such simplification. The map is a model.)
Given that you're averaging across the left-out variables, the parameters for the variables you include in your model will be distributions, not point values.
That's only part of the Bayesian philosophy -- I'm ignoring theoretical uncertainty, measurement uncertainty, priors, etc -- but it seems to me that the idea that your parameters have distributions makes intuitive sense, in the same way that descriptive statistics have a distribution.
But do Bayesians theoretically acknowledge that there is one true fixed parameter value out in the 'real world?'
In my opinion, the answer is yes. There is an unknown value $\theta_0$ of the parameter and the prior distribution describes our knowledge/uncertainty about it. In the Bayesian mathematical modelling, $\theta_0$ is considered as the realization of a random variable following the prior distribution.
If we go and couple Bayesianism with a deterministic universe (before you say anything with the word 'quantum' in it, humour me and recall that this is not physics.stackexchange) we get some interesting results.
Making our assumptions explicit:
Now, the deterministic universe may be one where atoms are newtonian little billiard balls. It may be entirely non-quantum. Let's say it is.
The agent now flips a fair coin. Think about that for a second, what does a fair coin constitute in a deterministic universe? A coin that has a 50/50 probability ratio?
But it is deterministic! With enough computing power you can calculate exactly how the coin will land, purely by simulating a model of a coin being flipped in the same manner.
In a deterministic universe a fair coin would be a disc of metal with uniform density. No force compels it to spend more time with one face down than the other (think about how weighted dice function.)
So the agent flips a fair coin. Yet, the agent is not quite powerful enough. It does not have sharp enough eyes to measure how the coin spins when flipped, it sees but a blur.
And so it says "This coin will land a heads with 50% probability." Lack of information leads to probabilities.
We may look at the phase space of how a coin is thrown. A large multidimensional coordinate system with axes pertaining to direction of throw, force of throw, spin of the coin, speed and direction of wind and so on. A single point in this space corresponds to a single possible coinflip.
If we ask the agent from before to colour in the coordinate system with a greyscale gradient corresponding to the agent's assignment of probability of heads for every given throw, it will most colour it all a uniform shade of grey.
If we the gradually give it more powerful internal computers with which to compute probabilities of heads, it will be able to make more and more discerning colourings. When we finally give it the most powerful internal computer, making it omniscient, it will effectively paint a strange checkerboard.
Fair coins are not made of probabilities, they are made of metal. Probabilities exist only in computational structures. So says the Bayesian.
I am not sure it is a relevant question because it requires more definition than the math itself requires. Because the math itself does not require it, I am not sure asking which Bayesian interpretation is correct has a lot of meaning.
Imagine two parallel universes. They are identical in the sense that the sequence of physical events in both universes unfolds in the same way.
In other words, of the sample space $\chi$, the Universe, $U\subset\chi$ is the same in every respect, $U_1=U_2$.
Now, in Universe One, every observer believes that Nature draws fixed points, $\theta_0$, from the parameter space at the start of time, $t=0$. These fixed points are $\theta_0\subset\Theta$. An observer denoted $i$, explains their initial uncertainty about its location with a probability distribution, $\pi_i(\theta_0)$ and upon seeing the data $X_i\subset{U}$ revises their uncertainty to $\pi(\theta_0|X_i)$.
Now, in Universe Two, every observer believes that Nature draws values for parameters, $\theta_t$, from a distribution which is believed to be approximated by $\pi_i(\theta_t)$ at time $t=\tau,$ which is when observer $i$ gathers the data. The draws are random. This differs from the concept of heteroskedasticity or stationary variables. Such a person would define either in a different manner. Upon seeing the data $X_\tau\subset{U}$ at time $\tau,$ they use this additional information to improve the description of that distribution of $\theta_t$ to $\pi_i(\theta_t|X_\tau)$.
Do note that in the second case, it isn't really helpful to bring in Frequentist definitions of things like time series, heteroskedasticity, or stationary variables because their ideas are predicated on fixed points. Also, there is nothing in Universe Two that prohibits the distribution from being a Dirac Delta function. However, nothing prevents a prior in Universe One from being one either, and as such, one could completely miss $\theta_0$.
If you drop the needless subsidiary notation, you end up with $\pi(\theta|X)\propto\pi(\theta)f(X|\theta)$.
The math provides no mechanism to be able to distinguish a world with fixed but unobservable parameters and a mechanism to describe that uncertainty from a world where the parameters truly are random variables.
Which is it? Who knows?
That Frequentist methods in some sense "work," doesn't provide a solution either. There is nothing about countably additive sets that makes them better than finitely additive sets. It is true that there are use cases where only a null hypothesis method or only a Bayesian method could possibly work. They are not the general case.
Interestingly, in those handfuls of cases where only one method could be thought of as suitable, the problem isn't resolved.
For example, if the critical element of your method boils down to a sharp null hypothesis such as $$H_0:\beta_1,\beta_2,\dots\beta_k=0$$ it does depend on on a mathematical conditioning of those parameters at zero, as if it were the true fixed point. Nature isn't required to listen. Indeed, if nature were sometimes drawing parameters instead and causing false positives or negatives, the method would not be able to tell.
Likewise, if the critical element of your method is setting gambling odds, there is no way to distinguish either world. You have to use a Bayesian method, but either conceptualization will work either well.
There are improper priors, for example Jeffreys, which has a certain relation to Fishers Information matrix. Then it is not subjective.
