# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

910 questions
81 views

### How to know whether a Gibbs sampler is irreducible? [duplicate]

How to know whether a Gibbs sampler is irreducible? I know that the Gibbs sampler in e.g. two variable case constructs a sequence of r.v.s $(X_1^{(i)}, X_2^{(i)})$ by sampling from the related ...
274 views

### Markov Chain: Simple Symmetric Random walk on {0,1,…,k}

Consider a simple symmetric random walk on {0,1,...,k} with reflecting boundaries. if the walk is at state 0, it moves to 1 on the next step. If the walk is at k, it moves to k-1 on the next step. ...
853 views

1k views

### Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
62 views

### Reinforcement Learning: 'Definition'/Construction of State and Action random variables

This is a follow-up on this question: In reinforcement learning, what is the formal definition of the symbols $S_t$ and $A_t$? I want to understand the construction and/or definition of the random ...
34 views

### Conditional entropy for Markov chain

Say you have a Markov chain $X \to Y \to Z$, and you want to compute $H(Z | X)$. Other quantities, such as $H(Z)$, $H(Y)$, $H(Z | Y)$ and $H(Y | X)$, are relatively straight-forward to compute. Is ...
53 views

62 views

### Explanation of the Markov chain in the given plot

I am trying to run a Markov process. I came across this webpage, which details a simple case of Markov process in R and python. While all the details are clear to me including 1) genotype of the ...
389 views

### Why does reward function depend on expected reward of next state?

I am confused about how could we get the reward scores of next state since we have not got into next state? I thought we will get the reward for the state we are now once we reached this state?
205 views

### Metropolis-Hastings Algorithm for Numerical Integration [duplicate]

I'm attempting to implement a Metropolis-Hastings Algorithm to evaluate integrals of the following form: $$I =\frac{1}{\sqrt\pi}\int_{-\infty}^{\infty} {f(x)\exp(-x^2)} \text{d}x$$ Now we can ...
58 views

### Appropriate method for time series data

I am interested in comparing the presence/absence of different species (MUVI80, MUXX80, ...
574 views

### Markov Chains vs. Bayesian Network

Background: I'm doing a project that takes a user's tweets and uses Markov Chaining to make up tweets. I'm putting together a presentation on this, and I'm struggling to make a meaningful distinction ...
268 views

### What is the relation and/or difference between Game Theory and Markov Chain Model? [closed]

I am doing some work regarding my master's thesis in networks security. I have decided to work with Game Theory, calculating the Nash Equilibrium for a two player zero sum game. However, I have also ...
646 views

### What if Markov chain does not converge in a reasonable amount of time?

I'm doing data analysis using Hamiltonian Monte Carlo for sampling from the posterior distribution of weights of a neural network. I'm using the Gelman-Rubin diagnostic estimated potential scale ...
65 views

### Project Euler Problem 213 (continued…) [duplicate]

This question is a continuation of How should one approch Project Euler problem 213 ("Flea Circus")? I followed the approach of Glen_b outlined in his answer in the page referenced above. I ...
431 views

### A Markov process has a Gaussian stationary distribution. What is implied about the tails of the conditional distribution?

Suppose that for all $t\in\mathbb{Z}$, the distribution of $x_t|x_{t-1},x_{t-2},\dots$ has probability density function $f(x_t|x_{t-1})$, where $x_t,x_{t-1}\in\mathbb{R}^n$. Suppose further that the ...
64 views

### How to compare likelihood of models which produce very small or zero probabilites?

I am trying to compute likelihood scores to measure the relative predictive accuracy of different hyperparameterisations of a complex statistical model against a set of discrete data. The model is a ...
57 views

### Finding the expected value of a random variable by simulating a Markov process

Suppose we have a pmf $\pi=\{\pi_1,\pi_2,...\}$ and a random variable $X$ such that $P(X=1)=\pi_1, P(X=2)=\pi_2,...$. I want to know how we can estimate $E(X)$. One way I know is (this is probably not ...
1k views

### Difference between a markov process and a semi- markov process [closed]

As the title suggest, i have a little problem grasping the main difference between markov and semi-markov processes. Anyone up for the challenge? :)
269 views

### Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
49 views

2k views

### Can somebody explain to me NUTS in english?

My understanding of the algorithm is the following: No U-Turn Sampler (NUTS) is a Hamiltonian Monte Carlo Method. This means that it is not a Markov Chain method and thus, this algorithm avoids the ...
288 views

### Irreducible Markov Chain Question

If you have an irreducible Markov chain with transition matrix $P$, and $p(j,j) > 0$ for all $j$, why are all states aperiodic?
96 views

### Most common Markov chain CLT for Metropolis-Hastings

Which Markov chain central limit theorem is the one most commonly used to justify standard error estimations in the Metropolis-Hastings algorithm? Is there one?
179 views

686 views

### Relationship between optimal action-value function and optimal value function

I would like to clarify on the relationship between the optimal action-value function of an MDP and the optimal value function as I often get confused between them. Is it possible to express one of ...