How can we assume the models are exhaustive in Bayesian Model Averaging? Bayesian model averaging is justified using the law of total probability which requires the the set of models that we average over to be exhaustive. Shouldn’t  we prove that the set of models are exhaustive first?
Even if you were to claim that this marginal density is an approximation; can w e prove that there exists a set of models which are exhaustive and the set of models that we are averaging in our approximation is a random subset of that?
 A: Clearly, such an assumption will realistically never hold (I'm sure there's some rare situations where it does - e.g. decay of atoms perhaps?). However, it may approximately hold (i.e. there's a couple of plausible models that might approximate the real data generating process pretty closely).
Another justification for model averaging is surely to give the most weight to the model that most closely approximates the data generating process? Burnham and Anderson argue that perspective in their "Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach" book.
A: The first thing here is that there are different concepts about what a model even means. Are we talking about a frequentist/aleatory interpretation of probability, where the model refers to a real process with repeatable observations, or are we talking about an epistemic interpretation of probability, in which a model refers to the uncertainty of an observer/researcher given certain knowledge?
In most Bayesian literature probabilities are epistemic, but many use Bayesian model averaging with a more frequentist idea in mind; some even mix the interpretations up, interpreting the lower level models in a frequentist manner, and the higher level distribution averaging over the models in an epistemic manner. The meaning of the question actually depends on this.
But even though this is so, my basic answer is the same in all cases. Models are idealisations and no model is ever meant to "hold" precisely. This applies both to frequentist and to epistemic models (although in epistemic modelling the modeller can basically choose to say that the model holds for them, as they are the one to be modelled; not that this solves many problems...).
The concept of "approximation" of reality by a model is not unproblematic either. Most notions of approximation are still based on assumptions that in reality are not true or not verifiable (e.g., the idea that an assumed cdf should approximate the empirical one assumes i.i.d. data and nothing in the world is really i.i.d. nor at least exchangeable when thinking about Bayesian assumptions).
If we assume a single model, we basically choose to look at the phenomenon of interest through the lens of a single model, and we need to be careful because we may miss something or may be led astray because of this. Averaging over a well defined set of models, our view is broader, allowing for several models to be considered, but this doesn't mean it is ever perfect or exhaustive. Ultimately the task is pragmatic: We're making model assumptions and we should always have in mind that these model assumptions will be violated in one way or another, and think about realistic ways in which this could happen. What is particularly important is to think about potential issues with the model that may mislead results (such as outliers, in many situations). This can lead to extension of the model class (acknowledging that even such an extension will never be exhaustive), to alternative approaches such as robust estimation, or to just to healthy caution when interpreting results.
One could say that averaging over several models is better because it gives us a better chance to get things "approximately right", however obviously this relies on further assumptions (regarding the prior), and it isn't progress if people who do this think that they need to be cautious with a single model, but when averaging models everything relevant can be safely covered.
A: This can be formally viewed as conditioning on another event $C$ that asserts (i.e., another modeling assumption) that the true model is one of the models under consideration.

For simplicity, let's assume that we are interested in finding $P(A)$ for some event $A$. A true exhaustive partition is given by $B_1, B_2, \ldots B_N, B_{N+1}$. The relevant analogy here, is that we are unaware of the event $B_{N+1}$ and we (incorrectly) assume that $B_1, \ldots B_N$ is exhaustive.
That is, we are implicitly conditioning on the event $C = \bigcup_{i=1}^N B_i$, which has probability $P(C) = 1 - P(B_{N+1})$. Conditional on $C$, the probability of each event in our "partition" is now
$$P(B_i|C) = \frac{P(B_i)}{1-P(B_{N+1})}.$$
So when we try to compute the probability of $A$, we are actually computing the probability of $A$ conditional on our modeling assumption $C$.
\begin{align*}
P(A|C) &= \sum_{i=1}^N P(A|B_i, C)P(B_i|C) \\[1.2ex]
&= \sum_{i=1}^N P(A|B_i, C)\frac{P(B_i)}{1-P(B_{N+1})} \\[1.2ex]
&= \frac{1}{1-P(B_{N+1})}\sum_{i=1}^N P(A|B_i, C)P(B_i)  \\[1.2ex]
&= \frac{1}{1-P(B_{N+1})}\sum_{i=1}^N P(A|B_i)P(B_i) && \text{since $B_i \cap C = B_i$} \\[1.2ex]
\end{align*}
From here, we can easily see that
$$P(A) = P(A|C)(1-P(B_{N+1})) + P(A|B_{N+1})P(B_{N+1})$$
Thus, as long as $P(B_{N+1})$ is sufficiently small, the modeling assumption will lead us to a reasonably good approximation for the target probability.

To relate this back to the original question (and also to the other answers), there are two questions we must consider.

*

*Are the models under consideration "nearly" exhaustive? (i.e. is $P(C) \approx 1$)

*Do the models not under consideration give very different answers from the models under consideration? (i.e. is $P(A|B_{N+1}) \approx P(A|C)$)

The answers to these questions will determine the success of BMA.
