Model averaging

We have the equation: $P(C|X)=\sum_{i}P(C|X,M_{i})P(M_{i}|X)$

I should technically be able to prove that it works even when the total number of models is 1, so I go from the right hand side of the equation

$P(C|X,M_{1})P(M_{1}|X)=\frac{P(C,X,M_{1})P(X,M_{1})}{P(X,M_{1})P(X)}=P(C,M_{1}|X)$

But I should be able to get P(C|X). Am I missing something?

• Try not removing the index $i$ when doing the calculcation. What does it mean when $i=1$? Think about it. Aug 23 '16 at 3:06
• I don't get it? i=1 means that I am considering only the first model (and in my case the only model). Aug 23 '16 at 3:16
• If you break a conditional probability into cases, then the union of all the cases you are breaking into must give you back the whole space. Aug 23 '16 at 3:19

You made a mistake in your calculation. You assumed $i=1$ but forgot that in order to use the law of total probability, you must have a countable partition of the sample space.

If $i=1$, then you must have $M=\Omega$ (here $\Omega$ is representing the sample space). Assuming that the collection $\{M_i\}_i$ is a partition of $\Omega$:

\begin{align} \sum_i P(C|X,M_i)P(M_i|X) &= \sum_i \frac{P(C,X,M_i)}{P(X,M_i)}\cdot\frac{P(M_i,X)}{P(X)}\\ &=\sum_i \frac{P(C,M_i,X)}{P(X)}\\ &=\sum_i P(C,M_i|X)\\ &=P(C|X) \end{align}

where the last equality follows from the partition.

• Hi, I just figured this out from your comment before you posted haha. Ultimately, it means that since there is just one model, C intersection M must just return C. Is that right? Aug 23 '16 at 3:19
• Yes. There is only one model. Aug 23 '16 at 3:20

Note: The posterior probablity:

$P(M_i|X)= \displaystyle\frac{P(X|M_i)P(M_i)}{\sum_kP(X|M_K)P(M_K)}$

So when i=1 we get that it's equal to 1, so if there is only one M, then it will happen for sure.

More inf on model averaging: http://www.stat.colostate.edu/~jah/papers/statsci.pdf