Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1
vote
0answers
81 views

How to know whether a Gibbs sampler is irreducible? [duplicate]

How to know whether a Gibbs sampler is irreducible? I know that the Gibbs sampler in e.g. two variable case constructs a sequence of r.v.s $(X_1^{(i)}, X_2^{(i)})$ by sampling from the related ...
2
votes
1answer
274 views

Markov Chain: Simple Symmetric Random walk on {0,1,…,k}

Consider a simple symmetric random walk on {0,1,...,k} with reflecting boundaries. if the walk is at state 0, it moves to 1 on the next step. If the walk is at k, it moves to k-1 on the next step. ...
3
votes
1answer
853 views

Why is the optimal policy non-stationary in the case finite-horizon problems, whereas it is stationary in the case of infinite-horizon problems?

I have difficulty understanding the meaning of stationary policy in the RL (MDP) setting. Specifically, let's assume stationary dynamics $$P(s_{t+1}=j|s_t=i,a) = P (s_{k+1}=j|s_k=i,a) \ \forall t,k,i,...
2
votes
1answer
667 views

Markov Property in practical RL

In the standard textbook RL setting we usually use the MDP framework where we assume that the current state is independent of the the whole history given the previous state. Obviously, in real life ...
1
vote
1answer
395 views

Long Run Proportion of Time in State of a Markov Chain

While reviewing for a stochastic processes exam, I came across the following proof in Introduction to Stochastic Processes with R by Dobrow. The proof is for the theorem that the expected number of ...
0
votes
1answer
40 views

Hidden Markov Chains - Is there a typo in this lecture?

Lecture in question: https://www.cse.buffalo.edu/~jcorso/t/CSE555/files/lecture_hmm.pdf Slide #6 shows this graph: Slide #7 gives these probabilities: Is P(Dry|High) = 0.3 on slide #7 a typo?
0
votes
0answers
231 views

Is this a Markov Chain and/or Poisson thing?

I know how to predict the winner of a football match in Python with a simple / multiple regression model, or Bayes. But I have the intuition that the last matches played are more relevant for the ...
2
votes
1answer
108 views

Grid search methods for posterior distribution approximation

I'm reading the book "Bayesian Analysis with Python" and the author provides some python code designed to show the grid search method of obtaining an approximate posterior distribution for the classic ...
1
vote
1answer
942 views

Markov Chains: Dice Problem I'm not sure how to start

The Problem: You start with five dice. Roll all the dice and put aside those dice that come up 6. Then roll the remaining dice, putting aside those dice that come up 6. And so on. Let $X_n$ be the ...
1
vote
0answers
43 views

Moment conditions for MS GARCH (Haas et al.)

I am doing some research on MS GARCH, specifically the model proposed by Haas et al., however I am really stuck on the moment conditions derived in the appendix of the paper. Right before equation ($...
2
votes
1answer
424 views

Regular Markov Matrix and Limiting Distribution

Regular Transition Matrix A transition matrix $P$ is said to be regular if some power of $P$ is positive. That is, $P^n >0$,for some $n≥1$. My question is: Do all regular matrices have limiting ...
0
votes
0answers
405 views

Second order markov tranision probability matrix

I tried to find the second order Markov chain of the following sequence Dat= A A B A B A A A B B A A B I tried it on "Markov chain" package in R. ...
0
votes
2answers
336 views

Markov-Chain transition probabilities for 3 variables

I am a bit confused as I need to calculate the Markov-Chain transition probabilites for 3 variables. Example data, let's assume a sequence of letters at specific and progressively-constant time steps:...
1
vote
1answer
193 views

Question on Detailed Balance and Bayes' Rule

I think I am confused due to the lax notation typically used when dealing with probabilities and not having a formal probability background. Bayes' Rule tells me that $$Pr(X_t=a|X_{t+1}=b)Pr(X_{t+1}=...
4
votes
1answer
1k views

Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
0
votes
0answers
62 views

Reinforcement Learning: 'Definition'/Construction of State and Action random variables

This is a follow-up on this question: In reinforcement learning, what is the formal definition of the symbols $S_t$ and $A_t$? I want to understand the construction and/or definition of the random ...
1
vote
0answers
34 views

Conditional entropy for Markov chain

Say you have a Markov chain $X \to Y \to Z$, and you want to compute $H(Z | X)$. Other quantities, such as $H(Z)$, $H(Y)$, $H(Z | Y)$ and $H(Y | X)$, are relatively straight-forward to compute. Is ...
0
votes
2answers
53 views

Proof that state at time $n+1$ depends only on the state at time $n$ using definition of Markov property

I'm learning about the Markov process in class and I have this step that is quite trivial that I don't know how to show. I also don't know how to go about googling this. I want to show that: $$P(X_{n+...
1
vote
1answer
69 views

Finding a mixture of 1st and 0'th order Markov models that is closest to an empirical distribution

I am interested in finding the distribution "$p^*$" closest to an empirical distribution $\hat{p}$ where $p^*$ is a mixture of first and zeroth order Markov models. That is, I want to find $$ p^* = \...
0
votes
1answer
62 views

Explanation of the Markov chain in the given plot

I am trying to run a Markov process. I came across this webpage, which details a simple case of Markov process in R and python. While all the details are clear to me including 1) genotype of the ...
1
vote
1answer
389 views

Why does reward function depend on expected reward of next state?

I am confused about how could we get the reward scores of next state since we have not got into next state? I thought we will get the reward for the state we are now once we reached this state?
0
votes
1answer
205 views

Metropolis-Hastings Algorithm for Numerical Integration [duplicate]

I'm attempting to implement a Metropolis-Hastings Algorithm to evaluate integrals of the following form: $$I =\frac{1}{\sqrt\pi}\int_{-\infty}^{\infty} {f(x)\exp(-x^2)} \text{d}x$$ Now we can ...
2
votes
1answer
58 views

Appropriate method for time series data

I am interested in comparing the presence/absence of different species (MUVI80, MUXX80, ...
0
votes
1answer
574 views

Markov Chains vs. Bayesian Network

Background: I'm doing a project that takes a user's tweets and uses Markov Chaining to make up tweets. I'm putting together a presentation on this, and I'm struggling to make a meaningful distinction ...
1
vote
1answer
268 views

What is the relation and/or difference between Game Theory and Markov Chain Model? [closed]

I am doing some work regarding my master's thesis in networks security. I have decided to work with Game Theory, calculating the Nash Equilibrium for a two player zero sum game. However, I have also ...
5
votes
2answers
646 views

What if Markov chain does not converge in a reasonable amount of time?

I'm doing data analysis using Hamiltonian Monte Carlo for sampling from the posterior distribution of weights of a neural network. I'm using the Gelman-Rubin diagnostic estimated potential scale ...
0
votes
0answers
65 views

Project Euler Problem 213 (continued…) [duplicate]

This question is a continuation of How should one approch Project Euler problem 213 ("Flea Circus")? I followed the approach of Glen_b outlined in his answer in the page referenced above. I ...
4
votes
2answers
431 views

A Markov process has a Gaussian stationary distribution. What is implied about the tails of the conditional distribution?

Suppose that for all $t\in\mathbb{Z}$, the distribution of $x_t|x_{t-1},x_{t-2},\dots$ has probability density function $f(x_t|x_{t-1})$, where $x_t,x_{t-1}\in\mathbb{R}^n$. Suppose further that the ...
0
votes
1answer
64 views

How to compare likelihood of models which produce very small or zero probabilites?

I am trying to compute likelihood scores to measure the relative predictive accuracy of different hyperparameterisations of a complex statistical model against a set of discrete data. The model is a ...
1
vote
0answers
57 views

Finding the expected value of a random variable by simulating a Markov process

Suppose we have a pmf $\pi=\{\pi_1,\pi_2,...\}$ and a random variable $X$ such that $P(X=1)=\pi_1, P(X=2)=\pi_2,...$. I want to know how we can estimate $E(X)$. One way I know is (this is probably not ...
1
vote
1answer
1k views

Difference between a markov process and a semi- markov process [closed]

As the title suggest, i have a little problem grasping the main difference between markov and semi-markov processes. Anyone up for the challenge? :)
3
votes
0answers
269 views

Convergence of approximate Gibbs sampling

We have a bivariate random variable $(X,Y)$ for which sampling is challenging. If we were to know how to sample from the conditionals $(X|Y)$ and $(Y|X)$, we could get samples from the joint using ...
1
vote
0answers
49 views

Question about Markov Chains

Is it true that for any real-valued Markov-chain $(X_n)_{n≥0}$ the sequence $(X_n+X_{n+1})_{n≥0}$ is again a Markov chain? I have started with the definition, I need to check that : $$P(Y_{n+1}=y_{n+1}...
1
vote
0answers
76 views

Dynamic Programming Planning Categories

Imagine a single player game that begins with six empty squares printed on a casino table. Each turn: The player pays a publicly visible, flat-rate cost (I'll call it "c") to take the turn The ...
0
votes
1answer
588 views

MLE for Markov Chains - intuitive explanation

could anyone please give me intuitive explanation what does below mean ? Let say I have sequence: a, b, a, b, b, b, a, b, b, a By applying Markov Chains with MLE method in R package on below I get ...
1
vote
0answers
46 views

Posterior Distribution for Markov Model

Let $X=\left\{x_{1},x_{2},x_{3},.....,x_{N}\right\}$ be a sequence of N observations where each $x_{i}$ is either 1 or 0. Now we are trying to fit a first order Markov model with the transition ...
0
votes
1answer
87 views

Applying MCMC Bayesian estimation to simple 2D problem

I'm having a conceptual problem applying Markov Chain Monte Carlo for Bayesian parameter estimation; I know how MCMC works in principle but it's applying it to this problem that troubles me. The ...
1
vote
0answers
56 views

Markov Chain MLE Covariance

 I'm trying to find the asymptotic covariance matrix $\Sigma$ for the MLE of a Markov Chain. Let $S$ be the states of the chain. If $\mathbf{y}$ is a path of the Markov Chain and $n_{ij}$ are ...
1
vote
0answers
120 views

What if transitions and emissions in a hidden Markov model are not independent?

A Hidden Markov Model is given by transition and emission matrices. The transition matrix determine probability of "next states" as a function of the current state. The emission matrix determine the ...
0
votes
1answer
64 views

Generating time series for electricity demand

I'm working on a project, trying to model electricity consumption and generation from a bunch of PV generators to see how much of the demand can be satisfied by the production in the "town". Most ...
1
vote
1answer
262 views

Number of parameters needed for a joint distribution when first order Markov chain is used

A joint probability distribution over D variables where each variable has K states, requires $ K^D $ parameters. How does the number of parameters needed reduce if the variables follow first order ...
3
votes
1answer
558 views

Gibbs sampling an Ising model with 0s and 1s

One of my problems in one of my courses ask to sample a 20 dimensional vector of 0s and 1s, $\{0,1\}^{20},$ when they are distributed as $$ \pi(x) = \exp\left\{-\beta \sum_{i=1}^{19} |x_{i+1}-x_i| \...
14
votes
2answers
2k views

Can somebody explain to me NUTS in english?

My understanding of the algorithm is the following: No U-Turn Sampler (NUTS) is a Hamiltonian Monte Carlo Method. This means that it is not a Markov Chain method and thus, this algorithm avoids the ...
0
votes
1answer
288 views

Irreducible Markov Chain Question

If you have an irreducible Markov chain with transition matrix $P$, and $p(j,j) > 0$ for all $j$, why are all states aperiodic?
5
votes
1answer
96 views

Most common Markov chain CLT for Metropolis-Hastings

Which Markov chain central limit theorem is the one most commonly used to justify standard error estimations in the Metropolis-Hastings algorithm? Is there one?
1
vote
0answers
179 views

Deriving Bellman Equation using optimal action-value function

How do we derive the associated Bellman equation from the optimal value function, $V^*(k)$ using the optimal action-value function which is $Q^*(k,a)$? Currently I've have/derived the following: $V^*(...
1
vote
1answer
155 views

Optimal policy of a Markov Decision Process

I'm not sure about the definition of optimal policy which is usually denoted by $\pi^*$. I'm not sure how we'd usually express optimal policy by, is it the best set of actions that gives the maximum ...
1
vote
1answer
686 views

Relationship between optimal action-value function and optimal value function

I would like to clarify on the relationship between the optimal action-value function of an MDP and the optimal value function as I often get confused between them. Is it possible to express one of ...
1
vote
1answer
553 views

“Symmetric” property of stationary distribution

The above symmetric property isn't referring to the double stochastic property \begin{bmatrix} 0.2&0.8&0&0&0\\ 0.2&0.2&0.6&0&0\\ 0&0.4&0.2&0.4&0\\ 0&...
1
vote
2answers
174 views

Hidden Markov Models with two emissions per state

I have a problem formulated in terms of hidden markov models. A simplified version is discussed here. I have a system that transitions between two discrete hidden states (states 1, 2). The transition ...