What does "Parameters are fixed and data vary" in frequentists' term and "Parameters vary and data are fixed" in Bayesians' term exactly mean? I hear the sentence in my question a lot, I kind of understand what it means but never have a clear picture of it. Hope to get the clear picture of what the sentence exactly mean.
 A: In frequentist philosophy, parameters are treated as non-random objects, while data are treated as random, hence "parameters are fixed and data vary".
In Bayesian philosophy, parameters are treated as random objects, and inference is performed by conditioning on an observed (fixed) set of data, hence "parameters vary and data are fixed". By parameters are treated as random objects, we mean parameters have a distribution, much like observations have distributions.
Note however, the interpretation is that this randomness reflects our belief of what the true underlying parameter is. In other words both Bayesians and frequentists agree that a true fixed parameter exists, but Bayesians further encode beliefs of what values this parameter might take on, in the form of a distribution.
To illustrate the difference in philosophies, consider an inference problem where we aim to construct an interval estimate for some parameter $\theta$ which is associated to the model by the sampling distribution whose density we denote as $f(X | \theta)$.
As a frequentist, you would infer a confidence interval, and a credible interval as a Bayesian.
Under the frequentist paradigm, you observe some data $X=x$ and construct a confidence interval by manipulating $x$, i.e., you have some function $C$ that maps $x$ to some interval. Because $X$ is a random variable, and $C$ is just a function of $X$, we are essentially constructing "random" interval estimates. The parameter is treated as a fixed, unknown constant. The meaning of confidence intervals is thus the probability of this random interval $C(X)$ capturing the fixed unknown constant $\theta$. Note this means if you observed say $100$ values of $x$, and you constructed a 95% confidence interval for each set of observations, you will capture $\theta$ for approximately $95$ of them.
Under the Bayesian paradigm, you begin by encoding your belief of what values the parameter might take on, say with a distribution $\pi_0$. Then you again observe some data $X=x$. To derive a credible interval you infer your updated belief,  encoded as a distribution called the posterior distribution, which we denote $\pi_1$. The posterior distribution is defined as
$$\pi_1(\theta | x) = \frac{f(x|\theta)\pi_0(\theta)}{p(x)}.$$
Here we see our posterior encodes our uncertainty of $\theta$ in the form of a distribution, much like how we encoded our belief prior to observing the data.
The data here is fixed in the sense that our estimate is conditioned upon what is observed. The credible interval is then taken as an interval of the posterior. The credible interval is interpreted as the probability of the parameter taking on values in the interval.
A: In Bayesian statistics we condition upon the observed data. The Bayesian part of your statement means that the data are known (and hence fixed to known values) and that the parameters are unknown (and hence allowed to vary and take on any plausible values).
In frequentist statistics, on the other hand, we compare the observed data to data that could have been observed. So we consider all sorts of hypothetical data (the data isn’t fixed). The fact that the parameters are fixed is a bit more nuanced. But essentially means that the results are not and do not use probability distributions for the unknown parameters. The frequentist results reduce to statements that assume specific values of the parameters or that are true for any assumed values. E.g., confidence intervals would reject the true parameters (whatever they are assumed to be) at a pre-specified rate.
