I've been trying to gain a better understanding of factor potentials in PYMC. In reading this article by Cam Davidson-Pilon on Yhat, I got confused about how observed nodes are understood by PYMC.
Are they fixed in PYMC when observed=True? Or are they random like a stochastic node?
The factor potential is specified as:
@mc.potential
def censorfactor(obs=obs):
if np.any((obs + birth < 10)[lifetime_.mask] ):
return -100000
else:
return 0
The variable birth and lifetime_ are fixed numpy arrays. Hence, it seems, the only changeable variable is obs. But I thought that was fixed in PYMC because it is defined by:
obs = mc.Weibull( 'obs', alpha, beta, value = lifetime_, observed = True )
If obs is fixed then the potential function will return a log probability of -100000 every time.
I suspect I'm missing a fundamental point of MCMC here. Appreciate if someone can help me understand.
UPDATE
I did some additional research and it appears that observed nodes can indeed be (partly) stochastic. If you pass a masked array as observed data to a stochastic node, each of the masked values will be individually represented as a stochastic node.
Chris Fonnesbeck's tutorial from this year's Scipy conference covers this topic at the end of this IPython notebook. Here he describes how the posterior predictive distribution is used to estimate each of the missing data points:
$$p(\tilde{y}|y) = \int p(\tilde{y}|\theta) f(\theta|y) d\theta$$
The ability to impute missing data conditional on the data and the model parameters (simultaneously) a powerful technique. Plus we get traces, distributions, HDI etc for each missing value.
