1
$\begingroup$

I am using MCMC with the Metropolos-Hasting algorithm to generate solutions of a non linear regression problem.

Likelihood

My likelihood is a gaussian distribution centered in 0 of the residuals with sigma as the measurements uncertainties.

Such as: $l(\theta|x)= \exp[ \frac{-\sum\limits_{i=1}^n (x_i-y_i(θ))^2}{ 2\sigma^2} ]=\exp[ -\chi^2 ]$

where $x_i$ is the observation i, $y_i(θ)$ is the prediction in function of the set of parameters θ, n is the number of observations, $\sigma$ is the measurements uncertainties.

Priors

The three discrete parameters can take any integer values between 1 and 20. There are the number of components used by three distincts PLS models to make predictions.

The three continuous parameters represents the average number of carbon atoms attached to three different functional groups. We know from past experiences that they are in certain ranges. So we have bounded uniform priors for these three parameters.

Proposal distribution

I am wondering how to set my proposal distribution, a multinormal distribution is not possible as I have discrete variables (3 continuous variables and 3 discrete).

I have thought about using uniform distributions for each variable as a proposal distribution. This won't be efficient but if I run enough iterations that should give meaningful results. Does anyone encountered a similar problem and has any advice?

Thanks for your time!

|cite|improve this question
$\endgroup$
  • $\begingroup$ Could you give more details on what your discrete variables look like? Preferably, describe the complete likelihood and prior. $\endgroup$ – Martin Modrák Feb 26 '18 at 15:46
  • $\begingroup$ @MartinModrák, yes I will add a description of the complete likelihood and priors. $\endgroup$ – Charlotte Feb 26 '18 at 15:49
1
$\begingroup$

This is not directly an answer for your concern rather a suggestion for a different approach. Given that each discrete variable has 20 levels, you may just be able to marginalize those 8000 combinations out (unless your data is large). In other words, instead of a discrete parameter, you have continuous parameters for the probability of each level (actually one parameter less as the probabilities have to sum to 1).

And if you are able to do this, you will be able to write your model in Stan, gain some efficiency and not worry about proposal distributions :-) See the "Latent discrete parameters" section of the Stan manual for more guidance and a discussion on why marginalizing might be good idea even if you use a different sampler.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Sorry for answering so late, I'm thinking about turning to Stan and marginalize out the discrete parameters. I looked at the Stan manual and the algebra, logic of marginalising makes a lot of sense. However, I didn't find any article/book except from the Stan manual where this technique is described. Do you have such a reference? $\endgroup$ – Charlotte Jun 5 '18 at 14:46
  • $\begingroup$ This book: bayesmodels.com supposedly has some examples, but I don't have a copy to check exactly what they say (the attached Stan code has those: github.com/stan-dev/example-models/tree/master/…). I've however seen people cite Stan manual frequently in publications - it is IMHO one of the best modelling resources out there. $\endgroup$ – Martin Modrák Jun 6 '18 at 7:17
  • $\begingroup$ I'm sorry to disturb you again, but in the documentation they showed how to calculate the posteriori distribution for the discrete parameters such as: $p(s|D) \propto \frac{1}{M} \sum_{m=1}^M (exp(lp[m,s]))$. I'm having trouble understanding why only the likelihood is taken into account and the prior for the continuous parameters is never taken into account. Am I missing something? $\endgroup$ – Charlotte Jun 27 '18 at 12:51
  • $\begingroup$ In Stan, you mostly write likelihood and priors separately, so my best bet is that they are discussing only likelihood and assume that users will specify priors as they see fit. $\endgroup$ – Martin Modrák Jul 3 '18 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.