A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms (1)

0
votes
0answers
11 views

how can we calculate the mean return time from state i to state j of markov chain in R?

I have calculated the mean state time from state i to i which is the reciprocal of stationary distribution E=1/pi where pi is stationary distribution in R can use the function ...
2
votes
0answers
22 views

Is there a survey that explores all the available Markov chain Monte Carlo methods?

I am interested in exploring the efficacy of various Monte Carlo methods. I am aware of the Metropolis acceptance criterion, Hamiltonian Markov chains, Gibbs sampling, importance sampling, slice ...
0
votes
0answers
10 views

How to make Markov Chain model from sequence of data using MATLAB? [on hold]

I have a sequence and from that I have to make Markov Chain Model in MATLAB. Markov Chain model considers only 1-step transition probabilities i.e. probability distribution of next state depends only ...
1
vote
1answer
35 views

Why is it necessary to fix a matrix diagonal and after this calculate the exponential to assess transition probabilities?

I'm learning markov chains in order to compute estimations of transition probabilities, and I found an example of the estimator construction for continuous time markov chains: ...
3
votes
0answers
30 views

What is the density of a markov chain when its transition probabilities have densities with respect to different measures?

I have a homogenous, discrete time Markov process, $(X_n)_{n\geq 0}$, with state space $\mathbb R_+$. Its transition probabilities have a density, $f(x_n\mid x_{n-1})$, with respect to the measure ...
0
votes
1answer
52 views

Convergence diagnostic of Markov chain that converge to uniform

Let $\Omega$ be a finite state space, $(X_t)_{t\in\mathbb{N}}$ be a discrete-time Markov chain that converges to the uniform distribution, and $P$ be its transition matrix. I'm looking for different ...
0
votes
1answer
52 views

Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
0
votes
0answers
21 views
+50

How to properly show the efficiency of a process?

I'm no statistician but my background is in computer science. At work, we are trying to improve the efficiency of a system where 5 people (A-E) each produce one part of a report and send it to 2 key ...
1
vote
2answers
37 views

Proof that Markov Property is not Satisfied at any Order?

My textbook has this figure in it: The textbook then says, Using d-separation, we can see there is always a path connecting $x_n$ and $x_{m}$ via the latent variables. This makes sense to me because ...
0
votes
2answers
22 views

Reinforcment learning MDP optimal policy existence

Reinforcement Learning: An Introduction. Second edition, in progress. Richard S. Sutton and Andrew G. Barto (c) 2012. Solving a reinforcement learning task means, roughly, finding a policy that ...
0
votes
0answers
31 views

Modeling the joint distribution of stream statistics

I have a question regarding computing the joint discrete probability distribution of statistics in a number stream. I posted this problem in the Mathematics section as well but I'm hoping the ...
1
vote
0answers
16 views

P value for Markov table

I have the following two-state Markov chain: ...
0
votes
0answers
28 views

Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1
vote
1answer
7 views

Question about marked poisson process

Let's say I have a Poisson point process on $\left[0,T\right]$ with rate $\lambda\left(t\right)=2t^2$. Suppose I attach a mark $m_t$ to each point $t$ of the process such that $m_t\sim ...
2
votes
1answer
36 views

Multiple-Try Metropolis question

I read Multiple-Try Metropolis from Wikipedia and I do not understand some points. Suppose the current state is $\mathbf{x}$. The MTM algorithm is as follows: Draw ''k'' independent ...
4
votes
3answers
83 views

How to generate the transition matrix of Markov Chain needed for Markov Chain Monte Carlo simulation?

I'm conducting a sensitivity analysis of a model using MCMC approaches. By reading the code of the sensitivity test procedure, I find the steps in Markov Chain is quite similar to random walk. Also, ...
0
votes
0answers
30 views

Estimating transition probability matrix for a given parameters

I am dealing with a hidden Markov model for variable $X_{t+1}$ where $X_{t+1}$ = $\alpha_{t}$$X_{t}$ + $(1-\alpha_{t})$$Z_{t}$ $X_{t}$ is an indication variable indicating whether an individual is ...
1
vote
0answers
15 views

The marginal likelihood of a fully-observed continuous time Markov chain

Say we have a fully observed trajectory $S$ from a CTMC. For a generator/rate matrix $Q$, we place gamma priors $\mathrm{Gamma}(\alpha_{1},\alpha_{2})$ on the diagonals, and ...
0
votes
0answers
27 views

Simulate probabilistic state machine

I have the following probabilistic state machine (psm): I need to simulate this psm, example of simulation can be: am, am. , do, ... Is it possible to use markov chain to simulate this psm? Col ...
0
votes
0answers
16 views

random walk on transition matrices

What kind of work has been done on modelling random walks on transition/stochastic matrices? Are there situations in which we can take two dirichlet random vectors, multiply them together, and then ...
1
vote
0answers
42 views

Correct way of computing Shannon Entropy of a walk

Take for example a walk such as: ["school", "work", "home", "kindergarten", "home", "school", ...] # or simply [1, 2, 3, 4, 3, 1, ...] What's the correct way of ...
1
vote
0answers
36 views

How can I use Monte Carlo to find a monthly premium?

The credit swap is as follows: I own a 100 million bond with an A rating. If that bond rating drops to a new low (during the ten years), I receive 20 million. I pay \$x a month for the arrangement. ...
1
vote
0answers
19 views

Can a Markov chain be approximated with an AR process?

In some MCMC literature/source code, a Markov chain is often approximated with an AR(1) process. There is some theory to suggest that such an approximation is somewhat valid for a finite state space, ...
0
votes
0answers
15 views

Kolmogorov Equations for a 3 state model. CTMC with a 3x3 generator matrix. Solving for $p_{11}$

I have a matrix $Q= \left[ \begin{array}{ccc} -3&3&0\\ 2&-5&3\\ 0&4&-4 \end{array} \right] $ where the state space is $S=[0,1,2]$ I need to solve the ...
4
votes
2answers
53 views

Number of Markov chain Monte Carlo Samples

There is a lot of literature out there about Markov chain Monte Carlo (MCMC) convergence diagnostics, including the most popular Gelman-Rubin diagnostic. However, all of these assess the convergence ...
0
votes
0answers
17 views

Is this a correct explanation of the markov assumption?

Here is a description of a the markov assumption (taken from http://di.ubi.pt/~jpaulo/competence/tutorials/hmm-tutorial-1.pdf) : Given W = word is this also a valid explanation of the markov ...
6
votes
1answer
93 views

Mars attack (probability to destroy $n$ spaceships with $k \cdot n$ missiles)

Suppose Earth has been attacked by $n$ Martian spaceships and suppose that we have $m=k \cdot n$ missiles to release against the $n$ spaceships. The probability to hit and destroy each spaceship by ...
2
votes
1answer
44 views

Calculating acceptance rate in Monte Carlo Markov Chain while doing Bayesian analyis

I am doing Bayesian analysis using a Monte Carlo Markov Chain of length 10000 and burn-in length 1000. I consider my chain as converged when the acceptance rate is equal to 23% and the chain mixing ...
4
votes
3answers
202 views

Probability of n-bit sequence appearing at least twice in m-bit sequence

Lets assume that we have a pattern $\alpha$ of bits of length $n$. Then I wish to know what the probability is of $\alpha$ appearing on a string of bits of length $m$ at least twice (where $m > ...
0
votes
0answers
13 views

Showing a queueing system is a Markov Chain

I generally understand how to do this but I'm having trouble with a formal proof. "Consider an $M/M/1/m+1$ queue with exponential arrivals rate $\lambda$, exponential service rate $\mu$, and finite ...
1
vote
0answers
17 views

EM vs. direct numerical optimization of likelihood function in high-dimensional Markov-Switching / HMM

I am currently estimating a Markov-switching model with many parameters using direct optimization of the log likelihood function (through the forward-backward algorithm). I do the numerical ...
1
vote
0answers
37 views

Learning a transition function of a Markov decision process using infrequently sampled data

I have a dataset of battery trials, where a battery has been charged and discharged in different regimes, and its capacity was measured periodically to reflect the effect of the charging and ...
2
votes
1answer
53 views

How to prove Markov Chain formula?

I saw an question about how often should I roll a 6-face dice to get every number at least once. I have read a lot of interesting answers but one of them caught my attention. It is one which imples ...
1
vote
0answers
13 views

simulating a stationary markov chain approach

I have hard time simulating a markov chain stated here: Consider a set of N transformations T= {t1,...,tn} . Let S={s1,...,sL} ...
0
votes
0answers
17 views

Markov vs Non Markov Process

I am a bit confused on this topic - I am aware that the a Markov Process can be adjusted to depend on more than just the preceding state...thus, don't all systems somewhat display Markov properties? ...
0
votes
0answers
11 views

Markov Birth Process Relating to Growth (non-stationary) - Need references, theory

I am looking for pointers into the literature for the problem described below so that I can build on what is already known. This problem fits into the category of a discrete time Markov pure birth ...
2
votes
1answer
50 views

Is a Markov chain with a limiting distribution a stationary process?

From Wikipedia: a stationary process (or strict(ly) stationary process or strong(ly) stationary process) is a stochastic process whose joint probability distribution does not change when ...
0
votes
1answer
47 views

Question about prior in bayesian image processing

I am learning Bayesian image processing. Bayesian approach will take prior knowledge about image into account. From one material, it says knowledge is expressed via probability functions. I understand ...
0
votes
0answers
26 views

Difference between linear regression and Markov analysis

What is the difference between linear regression and Markov analysis? As far as I understand both are used to predict future values. Can someone explain the difference (if possible using examples)?
0
votes
1answer
20 views

Using trigram language model to estimate probablilites

P(A | B) : probability of event A given event B occurred A second order Markov model is given as : Assume x0 = x-1=* in this definition where * is a special start symbol in the sentence. An ...
0
votes
0answers
30 views

Sparse markov chain

I have an instance (path) of discrete time Markov chain of length 10 millions observation with about 1.3 million of states. I am almost sure that the transition probability matrix will be very sparse. ...
3
votes
1answer
43 views

What is the distribution of the difference between two AR(1) processes?

I am reading a paper published in a good economics journal. An econometric model is presented in the paper. A part of the described model is not very clear to me. Please let me state a couple of ...
0
votes
1answer
20 views

Deriving transition matrix from infinitesimal generator, continuous time Markov chain

I' am reading Introduction to Stochastic Processes by Lawler and I' am a bit confused how demonstrates you get the transition matrix $\textbf{P}_t$ from the infinitesimal generator $\textbf{A}$. I'll ...
1
vote
1answer
54 views

Creating a transition matrix for markov chain

I have a dataset with monthly frequency of observations that fall in each category, Cat. I would like to construct a transition matrix from this, i.e., from ...
0
votes
1answer
14 views

Trying to understanding how finite-state space, continuous time Markov Chains are defined

I' am reading Introduction to Stochastic Processes by Lawler and am struggling to understand how continuous time, discrete state space processes are defined. Quote from the book, A (time-homogeneous) ...
2
votes
1answer
77 views

Unsure whether my continuous time Markov Chain distribution is correct

I' am reading Introduction to Stochastic Processes by Lawler and have hit problem 3.3(c) that I' am not sure I have correct. $\textbf{3.3}$ Suppose $X_t$ and $Y_t$ are independent Poisson processes ...
0
votes
0answers
19 views

Calculate state change probability

I am having telco order management data and need to calculate the probability of each order going through different stages. data is like this: Order No; product_type; time spent in step1; time ...
0
votes
0answers
24 views

Applying Markov Decision Processes to the Selling House Problem with waiting times

I'd like to apply the Markov Decision Process theory to this problem. We have a house to sell. Each day an offer of $X_n$ comes for the house. Each offer costs an amount $c$ to observe. You may think ...
2
votes
1answer
118 views

Which machine learning technique is appropriate for my problem?

I'm new in machine learning topics and I've problem in modeling my environment which has multi parameters with different value ranges and a few actions to perform when value of each parameter is not ...
3
votes
0answers
47 views

Testing Markov Property of transition matrix sequence with markovchain package in R

Related: Check memoryless property of a Markov chain I try to expand the verifyMarkovProperty function of the markovchain package to assess the Markov Property of transition matrices by assigning ...