# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,236 questions
Filter by
Sorted by
Tagged with
23 views

### Applying First order stochastic dominance to Markov Matrices [closed]

I'm interested in seeing if there is a concept like first order stochastic dominance but for Markov Matrices. The reason why is because I want to know if there is a way for us to define preference for ...
• 892
19 views

• 277
20 views

### Estimating markov transition matrix using total elevator "ups" and "downs" by floor

I have data on elevator presses and I am hoping to use them to estimate a Markov transition matrix, so I can ultimately estimate how frequently people go to different floors. For each floor from 1-4, ...
• 156
1 vote
14 views

### Estimating Markov Chain Probabilities with Limited Data

Suppose I have some data on transitions between states of a Discrete Time Markov Chain. Let's say that transitions between some events are observed more frequently from others. For example, in a 3 ...
543 views

### Markov Chains with Changing Number of States

I have seen these kinds of Discrete State Markov Chains before (Continuous Time or Discrete Time): Homogeneous (Probability Transition Matrix is constant) Non-Homogeneous (Probability Transition ...
• 91
64 views
+100

### How to tune the unadjusted Langevin algorithm?

I want to start investigating the (unadjusted) simulation of the Langevin process $${\rm d}X_t=b(X_t){\rm d}t+\sigma{\rm d}W_t,$$ where $$b:=\frac{\sigma^2}2\nabla\ln p.$$ I don't want to simulate ...
• 213
35 views

### Why reverse diffusion process is not a gaussian distribution?

The forward diffusion process, which goes from x_t to x_{t+1} is Gaussian, which is very reasonable as we go the next state by adding random gaussian noise. However, I do not understand why the ...
• 185
60 views

### Sum of powers (geometric series) of state transition matrix

I am working discrete time Markov chain analysis for some large state transition graph. I want to find the rewards/cost to reach from the init state to the terminal/accepting states. I have the state ...
• 101
7 views

### Which variable is best suited for edge weights when computing graph algorithms instead of relative risks?

I am currently trying to develop graph data. Which variable is best suited for edge weights when computing graph algorithms? Relative risk Relative Risk: Many networks in my field use relative risks ...
13 views

### Forward-Backward Algorithm for Autoregressive HMMs

I am currently studying HMMs, and covered the Forward-Backward Algorithms as well as the smoothing and filtering process. Recently, we were posed a question on Autoregressive HMMs which I've been ...
• 83
115 views

### msm package: Mutlti state model initial value in 'vmmin' is not finite

I am new to msm package and markov models. I have a randomized trial dataset with readings from three time points: baseline, at 1 year, and at 2 year. I am trying to calculate annual transition ...
160 views

### Can MCMC sample any probability distributions?

I have three fundamental questions related to MCMC. I would appreciate the help on any one of those. The most fundamental question in MCMC field, which I can't find a reference, is: Can MCMC generate ...
1 vote
48 views

• 213
10 views

### What is the effect of sampling rate on parameter estimation when fitting a markov state model to timeseries data?

Let us say that I have some timeseries data, which can be described by a markov state model. And the time series has been sampled every $\Delta t$ time units. The sampling rate ($1/\Delta t$) must ...
15 views

### How to test Markovian property in a financial time series?

I want to build a Markov Chain model for a financial time series to determine transition probabilities from one state to another. The underlying assumption is that series obeys the Markovian property. ...
• 487
33 views

### Help with Gambler's ruin problem, can't solve abstraction [duplicate]

I'm having difficulty solving this exercise. When I assume that p=0.4 and player A's fortune is 99 dollars and B's fortune is 1 dollar, I can find that the probability of player A losing to player B ...
1 vote
55 views

### Quesiton about markov chain period of transient

I got a quick question I got a markov chain with this trans matrix $$\begin{pmatrix}1&0\\1/2&1/2\end{pmatrix}$$ And I got 2 states right [0,1] right. So I know state 0 has a period 1 and is ...
37 views

### A recurrent Markov Chain implies its k-step version is also recurrent?

I am curious about whether a Markov Chain $X_n$ is recurrent implies that for any $k > 0$, $X_{kn}$ is also recurrent. Here are my observations. If $X_n$ is transient, $X_{kn}$ must be transient by ...
2k views

### Infinite dice roll probability

The following is an interview question: Two players A and B play a game rolling a fair die. If A rolls a 1, they immediately reroll, and if the reroll is less than 4 then A wins. Otherwise, B rolls. ...
• 491
441 views

### Deriving the Distribution of Markov Chain Times

I am interested in learning how to derive the probability distributions for the Time to Absorption in Markov Chains (Discrete and Continuous). In the past, I have usually done one of the following: ...
1 vote
Here is the paper link related to the question from my title. In Appendix B, it computes the entropy of $p(X^T)$ and says "By design, the cross entropy to $\pi(x^t)$ is constant under our ...