A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms (1)

1
vote
0answers
27 views

Randomly generating transition probabilities for Markov chains

I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ...
0
votes
0answers
30 views

'Lumpable states' analysis for a large transition matrix

I have a large transition matrix, whereby I calculate n-step state distribution results for n=1..10, and then merge states of interest for each ...
1
vote
0answers
23 views

From MDP to SMDP: What is it in a nutshell

Markov Decision Process (MDP) is a mathematical formulation of decision making. An agent is the decision maker. In the reinforcement learning framework, he is the learner or the decision maker. We ...
0
votes
0answers
38 views

Fast markov chain sampling of multiple initial states in R [migrated]

Assuming there are $K$ states. I would like to draw the next state for every initial state in a state vector 'initial_states' of length $N$ given a transition matrix 'trans_matrix' of dimension $K \...
4
votes
0answers
41 views

Markov Chain Monte Carlo (MCMC): How many samples are needed to get a uniform sample?

I am interested in a general answer although my question is rooted in a specific document. I am using the R package "hitandrun": https://cran.r-project.org/web/packages/hitandrun/hitandrun.pdf On ...
1
vote
2answers
106 views

Using Markov chains for prediction

i'm new to Markov theory and i'm trying to figure out how to solve the following question. Given the following transitions matrix (A), What is the probability of 5 consecutive Z?
3
votes
0answers
29 views

Optimal decision process to estimate Markov chain limiting distribution

Suppose there is a irreducible, reversible Markov chain with known states $1,\ldots,N$ and unknown transition matrix $T_{ij}$ and unknown limiting distribution $\pi_i$. I am able to repeatedly ...
1
vote
0answers
26 views

Bayesian MCMC Fitting

I am doing a Bayesian MCMC fit using emcee in python. I first maximize the log of the likelihood and use the results as initial parameter starting points in my MCMC. I am using a uniform prior and ...
0
votes
0answers
29 views

Nuisance Parameter in Bayesian MCMC

I am doing a Bayesian MCMC fit to some data using a simple model and I want to understand how to handle nuisance parameters. I am looking at this tutorial. The model is a line: $$y = m x + b$$. The ...
1
vote
1answer
34 views

Trouble in ERGODIC Markov chain

I don't fully understand ergodic. And I have trouble in this problem(21.8) from book introduction to information retrieval . Consider a Markov chain with three states A,B and C, and transition ...
3
votes
2answers
55 views

Transition probabilities - Markov chains

I have a homogeneous Markov chain with transition matrix I want to compute $P(Y_1 = 1| Y_2=2)$ where $Y_t, t=1,2$ is the observation at time $t$ and $Y_0=3$. I tried with Bayes' rule, so $$P(Y_1 = ...
0
votes
1answer
34 views

Trying to understand Markovian coupling

In http://www.cc.gatech.edu/~vigoda/MCMC_Course/MC-basics.pdf at one point it uses $P(X_0, \cdot)$ notation, but I don't understand what does it mean. I also don't understand how variation distance ...
1
vote
0answers
42 views

Significance testing for Markov chain transition probabilities

I'm modeling a sequence of discrete states as a first order, stationary Markov chain. I'd like to calculate a matrix of p values (one for each possible transition). Say the state at time $t$ is $S_t$. ...
3
votes
1answer
35 views

Interpretation of hidden states in HMM in the part-of-speech tagging task

Let me begin with a part-of-speech tagging task. The ultimate goal: given a sentence, what is the most probable part-of-speech tag for each word in the sentence? We want to answer this question by ...
1
vote
0answers
32 views

Metropolis algorithm to solve a problem [migrated]

I need to implement the metropolis algorithm to solve the example titled Cheating among students here. In summary the aim is to estimate the frequency of students cheating in an exam. The experiment ...
2
votes
1answer
27 views

Checking whether a given formula is correct for a homogeneous Markov chain

I am new to cross validated so I hope my question belongs here. I saw in a paper where I study someone claiming the following: Given a $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain (...
1
vote
1answer
50 views

What can I do with NA values in my second-order Markov chain?

I have states A, B, C, I have developed both a 1st and 2nd Order Markov Chain for them. Each state represents a status that an individual can be in, and the transitions represents the probability of ...
1
vote
0answers
29 views

Markov chain Monte Carlo sampling using CDFs instead of PDFs

I wonder if there is any MCMC sampling method which uses the definition of the target CDF instead of the target PDF; however, I may use a proposal PDF. I would like to use Metropolis-Hastings but it ...
0
votes
1answer
41 views

Steady State Calculation in Markov Chain in R

I am using the package markovchain in R. My transition matrix looks like this ...
1
vote
0answers
21 views

How is the Fermiac machine (Monte Carlo trolley) working?

There is a cool website showing the Markov chain with a machine. But nobody is explaining how it's working or showing a video of it's functioning. This is explaining the Markov chain monte carlo ...
4
votes
2answers
79 views

Use Markov chains to compute probability of rolling a 1 followed by a 2 before rolling two consecutive sixes

I was curious about how to apply the concept of Markov chains to the following problem: Rolling a die, what is the probability that 1 followed by 2 will happen before two sixes in a row? I have ...
4
votes
1answer
64 views

ergodic theory for markov processes

For an ergodic Markov Chain $$ \frac{1}{N}\sum_{i=1}^n f(X_i) \rightarrow E_\pi[f] $$ where $\pi$ is the invariant distribution. I am also dealing with a Markovian process (a state space model to ...
2
votes
1answer
39 views

Estimate transition matrix from many short Markov chains

I have a situation where data from the following process is observed: For $i = 1, \dots, n$ let $(X_{i,1}, \dots, X_{i,m_i})$ be a sequence of $m_i$ random variables coming from a discrete-space ...
12
votes
2answers
271 views

Sampling from an Improper Distribution (using MCMC and otherwise)

My basic question is: how would you sample from an improper distribution? Does it even make sense to sample from an improper distribution? Xi'an's comment here kind of addresses the question, but I ...
0
votes
0answers
20 views

In a markov chain, how to deal with final states that are not in the initial states - Probability of Default

I'm analyzing a bank portfolio in order to determine the consequent probability of default using markov chains. For this I select the group of credits at the end of month 'n' and measure their credit ...
0
votes
0answers
25 views

Markov Switching Model Forecasting

Let's say I have the following Markov switching model: $$r_t = 1.36 + a_t$$ $$a_t = \sigma_t \epsilon_t $$ $$ \sigma_t^2 = \left\{\begin{aligned} &0.15a_{t-1}^2 + 0.82\sigma_{t-1}^2 &&: ...
0
votes
0answers
37 views

Random walk as an example of time reversible Markov chains

My question is from the book Introduction to Probability Models, 10th edition, by Sheldon Ross. Here is a example in the book. [Page 250, Example 4.35] Consider a random walk with states $0, 1, . . ....
0
votes
1answer
40 views

Example of Conditional Expectation in Markov Chain

My question is from the book Introduction to Probability Models, 10th edition, by Sheldon Ross. Here is a example in the book. Consider a Markov chain with states $0, 1,\cdots , n$ having $P_{0,1} = ...
2
votes
0answers
57 views

Markov model parameter concentration and Fisher Information Matrix

For iid data, the posterior on the parameter $$ p(\theta \mid x_{0:T}) = \prod_{t=0}^T p(x_t \mid \theta) p(\theta) $$ is known to become independent of the prior which is the Bernstein-von Mises ...
0
votes
0answers
15 views

What are Linearly Solvable MDPs?

Markov Decision Process (MDP) is a formalism mainly used in artificial intelligence on the structure of decision making of a learner/agent. The aim is to find a suitable policy that maximizes the ...
0
votes
0answers
14 views

Revenue Forecasting using Markov Chain or Queuing Theory?

I am trying to forecast revenue for a HealthTech giant that sales HealthTech Hospital Equipment like Ultrasound, Magnetic resonance, CT AMI etc. The nature of business is Build to Order, which means ...
2
votes
2answers
34 views

Markov Decision Process - reinvesting rewards

I'm trying to find an optimal policy for a Stochastic game which I thought was simple, but seems increasingly complicated! It seems like this problem is accessible with a Markov Decision Process, but ...
0
votes
0answers
15 views

Kolmogorov Forward and Backward Equation Intepretation

Let $\lambda_i$ be the sojourn rate of state i, $q_{ij}$ be the transition rate form i to j, and $p_{ij}$ be the transition probability from i to j. The Kolmogorov Forward and backwards equation are ...
0
votes
0answers
27 views

how can we calculate the mean return time from state i to state j of markov chain in R?

I have calculated the mean state time from state i to i which is the reciprocal of stationary distribution E=1/pi where pi is stationary distribution in R can use the function E=1/steadyState(...
2
votes
0answers
36 views

Is there a survey that explores all the available Markov chain Monte Carlo methods?

I am interested in exploring the efficacy of various Monte Carlo methods. I am aware of the Metropolis acceptance criterion, Hamiltonian Markov chains, Gibbs sampling, importance sampling, slice ...
1
vote
1answer
48 views

Why is it necessary to fix a matrix diagonal and after this calculate the exponential to assess transition probabilities?

I'm learning markov chains in order to compute estimations of transition probabilities, and I found an example of the estimator construction for continuous time markov chains: http://www.rinfinance....
4
votes
0answers
40 views

What is the density of a markov chain when its transition probabilities have densities with respect to different measures?

I have a homogenous, discrete time Markov process, $(X_n)_{n\geq 0}$, with state space $\mathbb R_+$. Its transition probabilities have a density, $f(x_n\mid x_{n-1})$, with respect to the measure $\...
0
votes
1answer
55 views

Convergence diagnostic of Markov chain that converge to uniform

Let $\Omega$ be a finite state space, $(X_t)_{t\in\mathbb{N}}$ be a discrete-time Markov chain that converges to the uniform distribution, and $P$ be its transition matrix. I'm looking for different ...
0
votes
1answer
60 views

Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
0
votes
0answers
39 views

How to properly show the efficiency of a process?

I'm no statistician but my background is in computer science. At work, we are trying to improve the efficiency of a system where 5 people (A-E) each produce one part of a report and send it to 2 key ...
1
vote
2answers
46 views

Proof that Markov Property is not Satisfied at any Order?

My textbook has this figure in it: The textbook then says, Using d-separation, we can see there is always a path connecting $x_n$ and $x_{m}$ via the latent variables. This makes sense to me because ...
0
votes
2answers
27 views

Reinforcment learning MDP optimal policy existence

Reinforcement Learning: An Introduction. Second edition, in progress. Richard S. Sutton and Andrew G. Barto (c) 2012. Solving a reinforcement learning task means, roughly, finding a policy that ...
0
votes
0answers
31 views

Modeling the joint distribution of stream statistics

I have a question regarding computing the joint discrete probability distribution of statistics in a number stream. I posted this problem in the Mathematics section as well but I'm hoping the ...
1
vote
0answers
18 views

P value for Markov table

I have the following two-state Markov chain: ...
0
votes
0answers
29 views

Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1
vote
1answer
10 views

Question about marked poisson process

Let's say I have a Poisson point process on $\left[0,T\right]$ with rate $\lambda\left(t\right)=2t^2$. Suppose I attach a mark $m_t$ to each point $t$ of the process such that $m_t\sim N\left(t,1\...
2
votes
1answer
40 views

Multiple-Try Metropolis question

I read Multiple-Try Metropolis from Wikipedia and I do not understand some points. Suppose the current state is $\mathbf{x}$. The MTM algorithm is as follows: Draw ''k'' independent ...
4
votes
3answers
147 views

How to generate the transition matrix of Markov Chain needed for Markov Chain Monte Carlo simulation?

I'm conducting a sensitivity analysis of a model using MCMC approaches. By reading the code of the sensitivity test procedure, I find the steps in Markov Chain is quite similar to random walk. Also, ...
0
votes
0answers
35 views

Estimating transition probability matrix for a given parameters

I am dealing with a hidden Markov model for variable $X_{t+1}$ where $X_{t+1}$ = $\alpha_{t}$$X_{t}$ + $(1-\alpha_{t})$$Z_{t}$ $X_{t}$ is an indication variable indicating whether an individual is ...
1
vote
0answers
16 views

The marginal likelihood of a fully-observed continuous time Markov chain

Say we have a fully observed trajectory $S$ from a CTMC. For a generator/rate matrix $Q$, we place gamma priors $\mathrm{Gamma}(\alpha_{1},\alpha_{2})$ on the diagonals, and $\mathrm{Dirichlet}(\beta)...