A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms (1)

0
votes
0answers
7 views

Hidden Markov Model properties

I am dealing with state-space models in my course of Time Series Analysis and among others, I have read about the example of Hidden Markov Model that leaves me doubtful. I have studied that ...
1
vote
0answers
9 views

Checking whether a given formula is correct for a homogeneous Markov chain

I am new to cross validated so I hope my question belongs here. I saw in a paper where I study someone claiming the following: Given a $ \{ X_n \}_{n=0}^{\infty} $ be a homogeneous Markov chain ...
1
vote
1answer
44 views

What can I do with NA values in my second-order Markov chain?

I have states A, B, C, I have developed both a 1st and 2nd Order Markov Chain for them. Each state represents a status that an individual can be in, and the transitions represents the probability of ...
1
vote
0answers
26 views

Markov chain Monte Carlo sampling using CDFs instead of PDFs

I wonder if there is any MCMC sampling method which uses the definition of the target CDF instead of the target PDF; however, I may use a proposal PDF. I would like to use Metropolis-Hastings but it ...
0
votes
1answer
34 views

Steady State Calculation in Markov Chain in R

I am using the package markovchain in R. My transition matrix looks like this ...
1
vote
0answers
14 views

How is the Fermiac machine (Monte Carlo trolley) working?

There is a cool website showing the Markov chain with a machine. But nobody is explaining how it's working or showing a video of it's functioning. This is explaining the Markov chain monte carlo ...
4
votes
2answers
72 views

Use Markov chains to compute probability of rolling a 1 followed by a 2 before rolling two consecutive sixes

I was curious about how to apply the concept of Markov chains to the following problem: Rolling a die, what is the probability that 1 followed by 2 will happen before two sixes in a row? I have ...
4
votes
1answer
61 views

ergodic theory for markov processes

For an ergodic Markov Chain $$ \frac{1}{N}\sum_{i=1}^n f(X_i) \rightarrow E_\pi[f] $$ where $\pi$ is the invariant distribution. I am also dealing with a Markovian process (a state space model to ...
2
votes
1answer
35 views

Estimate transition matrix from many short Markov chains

I have a situation where data from the following process is observed: For $i = 1, \dots, n$ let $(X_{i,1}, \dots, X_{i,m_i})$ be a sequence of $m_i$ random variables coming from a discrete-space ...
12
votes
2answers
259 views

Sampling from an Improper Distribution (using MCMC and otherwise)

My basic question is: how would you sample from an improper distribution? Does it even make sense to sample from an improper distribution? Xi'an's comment here kind of addresses the question, but I ...
0
votes
0answers
18 views

In a markov chain, how to deal with final states that are not in the initial states - Probability of Default

I'm analyzing a bank portfolio in order to determine the consequent probability of default using markov chains. For this I select the group of credits at the end of month 'n' and measure their credit ...
0
votes
0answers
23 views

Markov Switching Model Forecasting

Let's say I have the following Markov switching model: $$r_t = 1.36 + a_t$$ $$a_t = \sigma_t \epsilon_t $$ $$ \sigma_t^2 = \left\{\begin{aligned} &0.15a_{t-1}^2 + 0.82\sigma_{t-1}^2 &&: ...
0
votes
0answers
36 views

Random walk as an example of time reversible Markov chains

My question is from the book Introduction to Probability Models, 10th edition, by Sheldon Ross. Here is a example in the book. [Page 250, Example 4.35] Consider a random walk with states $0, 1, . . ...
0
votes
1answer
38 views

Example of Conditional Expectation in Markov Chain

My question is from the book Introduction to Probability Models, 10th edition, by Sheldon Ross. Here is a example in the book. Consider a Markov chain with states $0, 1,\cdots , n$ having $P_{0,1} = ...
2
votes
0answers
55 views

Markov model parameter concentration and Fisher Information Matrix

For iid data, the posterior on the parameter $$ p(\theta \mid x_{0:T}) = \prod_{t=0}^T p(x_t \mid \theta) p(\theta) $$ is known to become independent of the prior which is the Bernstein-von Mises ...
0
votes
0answers
11 views

What are Linearly Solvable MDPs?

Markov Decision Process (MDP) is a formalism mainly used in artificial intelligence on the structure of decision making of a learner/agent. The aim is to find a suitable policy that maximizes the ...
0
votes
0answers
11 views

Revenue Forecasting using Markov Chain or Queuing Theory?

I am trying to forecast revenue for a HealthTech giant that sales HealthTech Hospital Equipment like Ultrasound, Magnetic resonance, CT AMI etc. The nature of business is Build to Order, which means ...
2
votes
2answers
31 views

Markov Decision Process - reinvesting rewards

I'm trying to find an optimal policy for a Stochastic game which I thought was simple, but seems increasingly complicated! It seems like this problem is accessible with a Markov Decision Process, but ...
0
votes
0answers
10 views

Kolmogorov Forward and Backward Equation Intepretation

Let $\lambda_i$ be the sojourn rate of state i, $q_{ij}$ be the transition rate form i to j, and $p_{ij}$ be the transition probability from i to j. The Kolmogorov Forward and backwards equation are ...
0
votes
0answers
24 views

how can we calculate the mean return time from state i to state j of markov chain in R?

I have calculated the mean state time from state i to i which is the reciprocal of stationary distribution E=1/pi where pi is stationary distribution in R can use the function ...
2
votes
0answers
32 views

Is there a survey that explores all the available Markov chain Monte Carlo methods?

I am interested in exploring the efficacy of various Monte Carlo methods. I am aware of the Metropolis acceptance criterion, Hamiltonian Markov chains, Gibbs sampling, importance sampling, slice ...
1
vote
1answer
43 views

Why is it necessary to fix a matrix diagonal and after this calculate the exponential to assess transition probabilities?

I'm learning markov chains in order to compute estimations of transition probabilities, and I found an example of the estimator construction for continuous time markov chains: ...
4
votes
0answers
37 views

What is the density of a markov chain when its transition probabilities have densities with respect to different measures?

I have a homogenous, discrete time Markov process, $(X_n)_{n\geq 0}$, with state space $\mathbb R_+$. Its transition probabilities have a density, $f(x_n\mid x_{n-1})$, with respect to the measure ...
0
votes
1answer
55 views

Convergence diagnostic of Markov chain that converge to uniform

Let $\Omega$ be a finite state space, $(X_t)_{t\in\mathbb{N}}$ be a discrete-time Markov chain that converges to the uniform distribution, and $P$ be its transition matrix. I'm looking for different ...
0
votes
1answer
58 views

Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
0
votes
0answers
36 views

How to properly show the efficiency of a process?

I'm no statistician but my background is in computer science. At work, we are trying to improve the efficiency of a system where 5 people (A-E) each produce one part of a report and send it to 2 key ...
1
vote
2answers
41 views

Proof that Markov Property is not Satisfied at any Order?

My textbook has this figure in it: The textbook then says, Using d-separation, we can see there is always a path connecting $x_n$ and $x_{m}$ via the latent variables. This makes sense to me because ...
0
votes
2answers
24 views

Reinforcment learning MDP optimal policy existence

Reinforcement Learning: An Introduction. Second edition, in progress. Richard S. Sutton and Andrew G. Barto (c) 2012. Solving a reinforcement learning task means, roughly, finding a policy that ...
0
votes
0answers
31 views

Modeling the joint distribution of stream statistics

I have a question regarding computing the joint discrete probability distribution of statistics in a number stream. I posted this problem in the Mathematics section as well but I'm hoping the ...
1
vote
0answers
16 views

P value for Markov table

I have the following two-state Markov chain: ...
0
votes
0answers
28 views

Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1
vote
1answer
9 views

Question about marked poisson process

Let's say I have a Poisson point process on $\left[0,T\right]$ with rate $\lambda\left(t\right)=2t^2$. Suppose I attach a mark $m_t$ to each point $t$ of the process such that $m_t\sim ...
2
votes
1answer
39 views

Multiple-Try Metropolis question

I read Multiple-Try Metropolis from Wikipedia and I do not understand some points. Suppose the current state is $\mathbf{x}$. The MTM algorithm is as follows: Draw ''k'' independent ...
4
votes
3answers
107 views

How to generate the transition matrix of Markov Chain needed for Markov Chain Monte Carlo simulation?

I'm conducting a sensitivity analysis of a model using MCMC approaches. By reading the code of the sensitivity test procedure, I find the steps in Markov Chain is quite similar to random walk. Also, ...
0
votes
0answers
32 views

Estimating transition probability matrix for a given parameters

I am dealing with a hidden Markov model for variable $X_{t+1}$ where $X_{t+1}$ = $\alpha_{t}$$X_{t}$ + $(1-\alpha_{t})$$Z_{t}$ $X_{t}$ is an indication variable indicating whether an individual is ...
1
vote
0answers
16 views

The marginal likelihood of a fully-observed continuous time Markov chain

Say we have a fully observed trajectory $S$ from a CTMC. For a generator/rate matrix $Q$, we place gamma priors $\mathrm{Gamma}(\alpha_{1},\alpha_{2})$ on the diagonals, and ...
0
votes
0answers
28 views

Simulate probabilistic state machine

I have the following probabilistic state machine (psm): I need to simulate this psm, example of simulation can be: am, am. , do, ... Is it possible to use markov chain to simulate this psm? Col ...
0
votes
0answers
18 views

random walk on transition matrices

What kind of work has been done on modelling random walks on transition/stochastic matrices? Are there situations in which we can take two dirichlet random vectors, multiply them together, and then ...
1
vote
0answers
66 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
1
vote
0answers
38 views

How can I use Monte Carlo to find a monthly premium?

The credit swap is as follows: I own a 100 million bond with an A rating. If that bond rating drops to a new low (during the ten years), I receive 20 million. I pay \$x a month for the arrangement. ...
2
votes
1answer
50 views

Can a Markov chain be approximated with an AR process?

In some MCMC literature/source code, a Markov chain is often approximated with an AR(1) process. There is some theory to suggest that such an approximation is somewhat valid for a finite state space, ...
0
votes
0answers
22 views

Kolmogorov Equations for a 3 state model. CTMC with a 3x3 generator matrix. Solving for $p_{11}$

I have a matrix $Q= \left[ \begin{array}{ccc} -3&3&0\\ 2&-5&3\\ 0&4&-4 \end{array} \right] $ where the state space is $S=[0,1,2]$ I need to solve the ...
4
votes
2answers
60 views

Number of Markov chain Monte Carlo Samples

There is a lot of literature out there about Markov chain Monte Carlo (MCMC) convergence diagnostics, including the most popular Gelman-Rubin diagnostic. However, all of these assess the convergence ...
0
votes
0answers
19 views

Is this a correct explanation of the markov assumption?

Here is a description of a the markov assumption (taken from http://di.ubi.pt/~jpaulo/competence/tutorials/hmm-tutorial-1.pdf) : Given W = word is this also a valid explanation of the markov ...
6
votes
1answer
95 views

Mars attack (probability to destroy $n$ spaceships with $k \cdot n$ missiles)

Suppose Earth has been attacked by $n$ Martian spaceships and suppose that we have $m=k \cdot n$ missiles to release against the $n$ spaceships. The probability to hit and destroy each spaceship by ...
2
votes
1answer
46 views

Calculating acceptance rate in Monte Carlo Markov Chain while doing Bayesian analyis

I am doing Bayesian analysis using a Monte Carlo Markov Chain of length 10000 and burn-in length 1000. I consider my chain as converged when the acceptance rate is equal to 23% and the chain mixing ...
4
votes
3answers
206 views

Probability of n-bit sequence appearing at least twice in m-bit sequence

Lets assume that we have a pattern $\alpha$ of bits of length $n$. Then I wish to know what the probability is of $\alpha$ appearing on a string of bits of length $m$ at least twice (where $m > ...
0
votes
0answers
14 views

Showing a queueing system is a Markov Chain

I generally understand how to do this but I'm having trouble with a formal proof. "Consider an $M/M/1/m+1$ queue with exponential arrivals rate $\lambda$, exponential service rate $\mu$, and finite ...
1
vote
0answers
21 views

EM vs. direct numerical optimization of likelihood function in high-dimensional Markov-Switching / HMM

I am currently estimating a Markov-switching model with many parameters using direct optimization of the log likelihood function (through the forward-backward algorithm). I do the numerical ...
1
vote
0answers
38 views

Learning a transition function of a Markov decision process using infrequently sampled data

I have a dataset of battery trials, where a battery has been charged and discharged in different regimes, and its capacity was measured periodically to reflect the effect of the charging and ...