# Tag Info

10

We can do this in a couple of simple ways. The first is easy to code, easy to understand and reasonably fast. The second is a little trickier, but much more efficient for this size of problem than the first method or other approaches mentioned here. Method 1: Quick and dirty. To get a single observation from the probability distribution of each row, we can ...

9

Standard programming practice: -when debugging run the simulation with fixed sources of randomness (i.e. same seed) so that any changes are due to code changes and not different random numbers -try your code on a model (or several models) where the answer IS known -adopt good programming habits so that you introduce fewer bugs -think very hard & long ...

9

It doesn't change the collinearity between the main effects at all. Scaling doesn't either. Any linear transform won't do that. What it changes is the correlation between main effects and their interactions. Even if A and B are independent with a correlation of 0, the correlation between A, and A:B will be dependent upon scale factors. Try this... a ...

9

Darren Wilkinson on his blog gives a detailed example of a common mistake in random walk Metropolis-Hastings. I recommend reading it in full, but here is the tl;dr version. If the target distribution is positive (like Gamma distributions etc) in one dimension, it is tempting to reject proposals that have a negative value on that dimension straight away. The ...

8

Have you considered using a nearest neighbour approach ? e.g. building a list of the k nearest neighbours for each of the 100'000 points and then consider the data point with the smallest distance of the kth neighbour a mode. In other words: find the point with the 'smallest bubble' containing k other points around this point. I'm not sure how robust ...

8

Yes the Handbook of MCMC is a very up-to-date collection of papers on MCMC, Also the book by Robert and Casella is a more current account than Markov Chain Monte Carlo in Practice. But I think MCMC in Practice is really a good place to start learning the subject. Here are amazon links to descriptions of the books I mentioned above. Introducing Monte Carlo ...

7

There is a nice and reasonably recent discussion of this problem here: http://www.icms.org.uk/downloads/mixtures/Robert.pdf Essentially, there are several standard strategies, and each has pros and cons. The most obvious thing to do is to formulate the prior in such a way as to ensure there is only one posterior mode (eg. order the means of the mixuture ...

7

I use the Gelman-Rubin convergence diagnostic as well. A potential problem with Gelman-Rubin is that it may mis-diagnose convergence if the shrink factor happens to be close to 1 by chance, in which case you can use a Gelman-Rubin-Brooks plot. See the "General Methods for Monitoring Convergence of Iterative Simulations" paper for details. This is ...

7

The short answer is yes. Have a look at sequential MCMC/ particle filters. Essentially, your prior consists of a bunch of particles ($M$). So to sample from your prior, just select a particle with probability $1/M$. Since each particle has equal probability of being chosen, this term disappears in the M-H ratio. A big problem with particle filters is ...

7

I believe that Weak convergence and optimal scaling of random walk Metropolis algorithms by Roberts, Gelman and Gilks is the source for the 0.234 optimal acceptance rate. What the paper shows is that, under certain assumptions, you can scale the random walk Metropolis-Hastings algorithm as the dimension of the space goes to infinity to get a limiting ...

7

Just to add to answer by @NRH. The general idea follows the Goldilocks principal: If the jumps are "too large", then the chain sticks; If the jumps are "too small", then the chain explores the parameter space very slower; We want the jumps to be just right. Of course the question is, what do we mean by "just right". Essentially, for a particular case ...

7

Unfortunately, harmless seeming priors can be very dangerous (and have even fooled some seasoned Bayesians). This recent paper, provides a nice introduction along with plotting methods to visualize the prior and posterior (usually marginal priors/posterior for the parameter(s) of interest). Hidden Dangers of Specifying Noninformative Priors. John W. ...

7

I'm not an expert in any of these, but I thought I'd put them out there anyway to see what the community thought. Corrections are welcome. One increasingly popular method, which is not terribly straightforward to implement, is called Hamiltonian Monte Carlo (or sometimes Hybrid Monte Carlo). It uses a physical model with potential and kinetic energy to ...

6

RJMCMC was introduced by Peter Green in a 1995 paper that is a citation classic. He wrote a Fortran program called AutoRJ for automatic RJMCMC; his page on this links to David Hastie's C program AutoMix. There's a list of freely available software for various RJMCMC algorithms in Table 1 of a 2005 paper by Scott Sisson. A Google search also finds some ...

6

The Arellano-Bond estimator has been designed for precisely this type of problems. You will find a short non-technical paper with a examples here. In a nutshell, it combines the information embedded in the large number of cross-section to make up for the small number of points in each series. This estimator is widely used and implemented: it is avalaible in ...

6

Gilles Celeux also worked on the problem of label switching, e.g. G. Celeux, Bayesian inference for Mixture: the label switching problem. Proceedings Compstat 98, pp. 227-232, Physica-Verlag (1998). As a complement to @darrenjw's fine answer, here are two online papers that reviewed alternative strategies: Jasra et al., Markov Chain Monte Carlo ...

6

So the simple answer is yes: Metropolis-Hastings and its special case Gibbs sampling :) General and powerful; whether or not it scales depends on the problem at hand. I'm not sure why you think sampling an arbitrary discrete distribution is more difficult than an arbitrary continuous distribution. If you can calculate the discrete distribution and the ...

6

The ideal Monte Carlo algorithm uses independent successive random values. In MCMC, successive values are not independant, which makes the method converge slower than ideal Monte Carlo; however, the faster it mixes, the faster the dependence decays in successive iterations¹, and the faster it converges. ¹ I mean here that the successive values are quickly ...

6

I find it surprising that a flat likelihood produces convergence issues: it is usually the opposite case that causes problems! The usual first check for such situations is to make sure that your posterior is proper: if not it would explain for endless excursions in the "tails". If the posterior is indeed proper, you could use fatter tail proposals like a ...

6

For a handbook and an extensive coverage, the following one is very moderately priced. Brooks, et al. (ed.), Handbook of Markov Chain Monte Carlo, Chapman & Hall/CRC, 2011. Robert and Casella (2010) have a good deal of theory.

6

I'm not sure whether this is exactly what you're after, but a couple of articles I've found useful on theoretical properties of various Metropolis-Hastings algorithms are: Optimal scaling for various Metropolis-Hastings algorithms - Roberts & Rosenthal, 2001. (This summarises some earlier results for the Ransom walk Metropolis and the ...

6

Parameters in linear predictor are t-distributed. When the number of records goes to infinity, it converges to normal distribution. So yes, normally it is considered correct to assume normal distribution of parameters. Anyways, in bayesian statistics, you need not to assume parameter distribution. Normally you specify so called uninformative priors. For ...

6

There are several diagnostics, including the Geweke Diagnostic, the Heidelberg and Welch Diagnostic, the Raftery and Lewis Diagnostic, and the Gelman and Rubin Multiple Sequence Diagnostic. Also, visual examination of the trace plot can help. All of these are only indications, not guarantees. You might check out: ...

6

Firstly, feel free to ask questions like this on our users' list (http://mc-stan.org/mailing-lists.html) where we discuss not only issues related to Stan implementations/optimizations/etc but also practical statistical and modeling questions. As to your question, it's absolutely a fine approach. There are many ways to justify it more rigorously (for ...

5

You can always choose one or several real valued functions of the categorical variables and look at the auto-correlation for the resulting sequence(s). You can, for instance, consider indicators of some subsets of the variables. However, if I understood your question correctly, your sequence is obtained by an MCMC algorithm on the discrete space. In that ...

5

Kolmogorov Smirnoff is always a good test to see if an arbitrary distribution fits. You can use the test cited below to see if two sets of data came from the same distribution: Li, Q. and E. Maasoumi and J.S. Racine (2009), “A Nonparametric Test for Equality of Distributions with Mixed Categorical and Continuous Data,” Journal of Econometrics, ...

5

Bayesian Statistics does not depend on Markov chains (well in theory), Markov chain Monte Carlo is a method for making the computations in Bayesian Statistics easier (doable). Generally we want to generate data from the posterior distribution which we can easily compute parts of, but not always all of (the normalizing coefficient is usually the hard part), ...

Only top voted, non community-wiki answers of a minimum length are eligible