I would recommend reading the underlying papers that this paper is derived from as the terminology doesn't appear to have become standard in the field. The original paper is by Rubin, but Gelman is writing from Meng.
Meng, X. (1994). Posterior Predictive p-Values. The Annals of Statistics, 22(3), 1142-1160.
As to your questions:
I am trying to interpret what is meant is meant when he says 'model is true'. My questions are:
i) Statistically, what is a "true model" as said in the quote above?
ii) What does a value of 0.5 mean in simple words?
So there is some unfortunate language usage as p-values are a Frequentist idea and Bayesian methods do not have p-values. Nonetheless, within the context of the articles beginning with Rubin, we can discuss the idea of a Bayesian p-value in a narrow sense.
As to question one, Bayesian models do not falsify a null hypothesis. In fact, except where some method is intending to mimic Frequentist methods, as in this paper, the phrasing "null hypothesis" is rarely used. Instead, Bayesian methods are generative methods and are usually constructed from a different set of axioms.
The easiest way to approach your question is from Cox's axioms.
Cox, R. T. (1961). The Algebra of Probable Inference. Baltimore, MD: Johns Hopkins University Press.
Cox's first axiom is that plausibilities of a proposition are a real number that varies with the information related to the proposition. Notice the word probability wasn't used as this also allows us to think in terms of odds or other mechanisms. This varies very strongly from null hypothesis methods. To see an example, consider binary hypotheses $H_1,H_2$, which in Frequentist methods will be denoted $H_0,H_A$. What is the difference?
$H_0$ is conditioned to be true and the p-value tests the probability of observing the sample, given the null is true. It does not test if the null is actually true or false and $H_A$ has no form of probability statement attached to it. So, if $p<.05$, this does not imply that $\Pr(H_A)>.95$.
In the Bayesian framework, each proposition has a probability assigned to it so that if $H_1:\mu\le{k}$ and $H_2:\mu>k$, then it follows that if $\Pr(H_1)=.7327$ then $\Pr(H_2)=.2673$.
The true model is the model that generated the data in nature. This varies from the Frequentist method which depends only on the sampling distribution of the statistic, generally.
As to question two, Gelman is responding to Meng. He was pointing out that in a broad variety of circumstances if the null hypothesis is true, then $\Pr(y^{rep}|y)$ will cluster around .5 if you average over the sample space. He provides a case where this is useful and one where it is a problem. However, the hint as to why comes from the examples, particularly the first.
The first has been constructed so that there are great prior uncertainty and this use of a nearly uninformative prior propagates through to the predictive distribution in such a way that, almost regardless of your sample, Rubin and Ming's posterior predictive p-values will be near 50%. In this case, it would mean that it would tell you there is a 50% chance the null is true, which is highly undesirable since you would rather be either near 100% or in the case of falsehood, 0%.
The idea of Bayesian posterior p-values is the observation that since you are now in the sample space as random, rather than the parameter space as random, the rough interpretation of a Bayesian posterior probability is remarkably close to the Frequentist p-value. It is problematic because the model is not considered a parameter, in itself, and has no prior probability as would be the case in a test of many different models. The model, $H_A$ versus $H_B$ is implicit.
This article is a warning of something that should, in a sense, be obvious. Imagine you had fifty million data points and there was no ambiguity as to the location of the parameter, then you would be stunned if the resulting predictive distribution was a bad estimator over the sample space. Now consider a model where the results are ambiguous and the prior was at best weakly informative, then even if the model is true, it would be surprising to get a clear result from the posterior predictive distribution.
He provides an example where data is drawn from a population that has a standard normal distribution. The required sample would have to be 28,000 to get a rejection of the model. In a standard normal population, that will never happen.
The model is about the propagation of uncertainty and whether or not Rubin/Meng's idea generates a useful construct when it is needed most when the data is poor, small, weak or ambiguous as opposed to samples that are stunningly clear. As an out-of-sample test tool, its sampling properties are undesirable in some circumstances, but desirable in others.
In this case, what Gelman is saying is that regardless of the true probability of the model, the out-of-sample validation score provided by the Bayesian posterior predictive p-value will be near 50% when the null is true when the data doesn't clearly point to a solution.
This has lead to the criticism of the idea as uncalibrated with the true probabilities. See
Bayarri, M. J. and Berger, J. (2000). P-values for composite null models. Journal of the American Statistical Association 95, 1127–1142.
