# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

924 questions
Filter by
Sorted by
Tagged with
17 views

### can the prior probability change with time?

I was reading Markov Models for sequence modeling and stuck with my understanding(hypothesis). In Baye's theorem, can the prior probabilities change with time? If the prior probability at t=3 is 0.05,...
18 views

### Inference and simulation on Markov Chain [on hold]

If I want to do inference for the next state on Markov Chain, do I need to find state staionary distribution first and then I do inference? Or state stationary distribution has nothing to do with ...
16 views

### Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
12 views

### Markov Chain Monte Carlo for Decryption Purposes

I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
16 views

### What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?

I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
6 views

### What are the relationships among Markov Property, Stationarity, and Time Invariance

I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...
19 views

### How to train a single discrete-time markov model?

I have a training set of sequences. I want to reach a discrete time Markov model (transition probability matrix). Is there a Bayesian way other than MLE to achieve this?
21 views

### How to estimate Markov Chain probabilities from a given NCD dataset by using MLE in R [closed]

I have a dataset about no claim discount(NCD)car insurance scheme with 10,000 policyholders in the current policy year. the question is to estimate the probability transition matrix for the Markov ...
50 views

### Spread of number of steps to reach absorbing state in markov chain

I know how to calculate the variance of the number of steps in an absorbing markov chain. However, I am not sure that the distribution of the number of steps is normal. Therefore I would like to ...
19 views

### Creating the input for mcl algorithm

I would like to implement mcl (Markov clustering algorithm) in a dataset like this one: ...
15 views

### How to find exact month of regime shifts in Markov switching model in R?

I'm looking into industry specific merger waves. I have used the Markov switching model (AR(0)) to identify periods with high M&A activity (merger wave periods) and periods with lower/normal M&...
88 views

### Understanding Convergence of Transitional Probabilities via Krylov Bogolivob Theorem

Could anyone break and write a few more lines for me of the proof of the Theorem 1.10 here, by Prof. Martin Hairer, on page 7 in particular. I am not understanding what he means by (1) the continuity ...
33 views

### Applicability of Markov Models for predicting user input

I am trying to predict user action based on the shown content of different modules. Lets assume the user sees a page where several content may be shown or not. The next action of the user depends on ...
43 views

12 views

I would like to use (hidden) Markov Chain to predict $X[t+1]$ stock price. Historical data for top biggest 500 companies for last 10 years will be used for training. The error would be sum of $... 1answer 55 views ### Metropolis-Hastings - interpreting the transition kernel: alpha*proposal I thought I had great intuition and mathematical understanding of the Metropolis-Hastings algorithm, until closer inspection... as I started compiling my notes, I realized I do not understand the ... 0answers 25 views ### Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions I want to sample from a distribution$p(X)$with$X \in R^n$. However, I can only evaluate the likelihoods of$Z = AX$and$Z = BX$with$A,B \in R^{m \times n}$and$m = n-1$. Now my idea is to run ... 0answers 11 views ### Zero Sum problem with MDP formulation and the difference with minimax approach Suppose that we have formulate a zero sum game with MDP and Ua(s) and Ub(s) are the utilities of A and B in the s state. Suppose that all rewards and utilities are calculated from the A's point of ... 1answer 32 views ### How can I compute$P(X_1 = 1|X_0 = 1)$from the given data? I know how to mathematically calculate the probability of various Markov Properties. But, how can I calculate Markov probability from data? Suppose, I have a Markov Chain as follows:$S=\{1, 2\}$... 0answers 32 views ### Understanding Approximate Dynamic Programming I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ... 1answer 29 views ### Markov Blanket of two nodes? I'm trying to solve a question and it has asked for (I feel like I'm confused by everything, would apprecaite some help, thanks), the Markov blanket of {c,d}. From what I've read so far, Markov ... 1answer 25 views ### MCMC changing multiple parameters What is the standard way of drawing multiple parameters for a MCMC run? Say I have 9 parameters, what is the most efficient method to get all parameters to explore their distributions properly? 1answer 40 views ### What happens if the observations are connected in a hidden Markov model (HMM)? Suppose that we have an HMM with hidden variables$X_t$and observed variables$Y_t$. Why do we always assume$p(Y_t|X_t)$? What happens if we have$p(Y_t|X_t, Y_{t-1})$? Is it because that wouldn't ... 1answer 106 views ### Manual simulation of Markov Chain in R Consider the Markov chain with state space S = {1, 2}, transition matrix and initial distribution α = (1/2, 1/2). Simulate 5 steps of the Markov chain (that is, simulate X0, X1, . . ... 1answer 32 views ### Hidden Markov Model probability of producing a sequence Suppose that we have two models for a 2-state HMM and both have two output symbols:$A$and$B$. Model 1: Transition probabilities:$a_{11}=0.6$,$a_{12}=0.4$,$a_{21}=0.0$,$𝑎_{22}=1.0$. Output ... 1answer 57 views ### For a Markov Chain is$𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$equals to$𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$? I am new to Markov Chains and using this concept in statistics. For a Markov Chain, may I say that$𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$equals to$𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$? If yes, how can I prove that? 1answer 44 views ### limit and stationary distribution of a Markov chain Consider a Markov chain on the non-negative integers with transition probabilities 􏰀$1/2$if$y=x+1$and$1/2$if$y=0$. Find$\lim_{n \to \infty} P(X_{n}=0)$. Is this limit the same as the ... 0answers 12 views ### Countably infinite state space for Continuous Time Markov Chains I have been unsuccessfully browsing the internet for resources related to Continuous Time Markov Chains with a countably infinite state space, for example$\mathbb{N}$. Is there any developed theory ... 2answers 89 views ### Why does Judea Pearl call his causal graphs Markovian? In his texts on causality, Judea Pearl always refers to the simplest graphs he uses, i.e. the acyclic graphs with independent confounders, as Markovian. I don't see why these graphs contain anything ... 1answer 28 views ### Compute the Limiting Distribution Consider the transition matrix$ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $for general$2$-state Markov Chain$(0 \le p, q\le 1)$. Find the limiting distribution ... 0answers 12 views ### Model selection with this model of a large number of components I have a discrete time Markov Chain$\{X_n: n \in \mathbb{N}_0\}$with unknown transition matrix$P \in \mathbb{R}^{M \times M}$on the state space$\mathcal{S}_X = \{1,2, \dots, M\}$, with$M \geq 2$.... 1answer 23 views ### How can I compute expected return time of a state in a Markov Chain? I was watching a YouTube video regarding the calculation of expected return time of a Markov Chain. https://www.youtube.com/watch?v=X_Ll0-Ytu7U&vl=en I haven't understood the calculation of$m_{...
I am reading the queueing theory volume 1 by Kleinrock. In the chapter on Continuous Time Markov Chain(CTMC), the author defines the infinitesimal generator, $Q(t)$ as having the following elements: \$...