# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,164 questions
Filter by
Sorted by
Tagged with
10 views

### Logistic regression(with Markov chain) on time series data?

I'm working with a biotech device's time series data to predict the replacement amount. The background is the battery of the device will die after the implant for a few years, and the battery will be ...
19 views

### Creating synthetic data for time series, Hidden Markov Model

Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
• 269
1 vote
18 views

### Markov chain: inferring transition rates from equilibrium

I feel like I have been having a very dumb week trying to solve/research this problem and that I am missing an easy solution or that it is not possible. Given an equilibrium distribution (say from a ...
90 views

### Why do we sample from the uniform distribution in Metropolis-Hastings for acceptance?

For each iteration of the MH, sample $x'=q(x|x')$, then the acceptance probability is computed:$$A=\min(1,a)$$ where $$\alpha=\frac{p(x')q(x|x')}{p(x)q(x'|x)}$$ Now, I've seen that the algorithm ...
• 487
1 vote
18 views

### How to prove a sequence of random variables dependent on different random variables is a homogeneous Markov chain?

In Pierre Brémaud's book, Markov Chains - Gibbs Fields, Monte Carlo Simulation and Queues, exercise 2.6.9 is stated as follows: Let $\{Z_n\}_{n \geq 1}$ be an IID sequence of geometric random ...
• 648
1 vote
69 views

### "SVD did not converge" while using statsmodel.timeseriesanalysis.MarkovRegression

I am trying to fit a MarkovRegression model sm.tsa.MarkovRegression(spread, k_regimes=2, switching_variance=True) to a time series of price spread. This is the ...
17 views

### Is this Markov chain of a UAU-process (unaware - aware - unaware) convergent and why?

I am currently looking at a Markov chain of $UAU$-process on a uni-weighted undirected network. Where individuals are aware of certain arbitrary information or not. The individuals are the nodes of ...
14 views

### For markov transition matrix and the initial state, calculate probability to reach certain other state in k or less steps

So there is a vector n giving the initial state and a Markov transition probability matrix M. I know I can calculate the ...
• 71
21 views

### Stationary Distributions and Financial Markets

Not sure my idea makes sense but here goes. Financial markets are notoriously hard to model. Is this perhaps there does not exist a stationary distribution for any Markov chain stochastic process ...
63 views

### Understanding the Difference Between Different Types of Markov Chains?

I have been trying to learn more about different types of Markov Chains. So far, here is my basic understanding of them: Discrete Time Markov Chain: Characterized by a constant transition probability ...
• 6,048
1 vote
32 views

### What is the influence of initial state in sequence generated from a markov chain?

For thousands of item, I have observations about their state (a letter) for 9 timestep. From that, I build a transition matrix (RotationMatrix by couting their ...
67 views

• 1,118
14 views

### Correlation of entries of the fundamental matrix of an absorbing Markov chain

Suppose I have an absorbing Markov chain with state space $S$, partitioned into $T$, the set of transient states, and $A$, the set of absorbing states. Let $N$ denote the fundamental matrix of this ...
• 1,002
72 views

### Why not just run a Markov chain to get stationary probabilities?

I'm reading Performance Modeling and Design of Computer Systems which contains some analysis of Markov chains. In particular, it emphasises various analytical methods for finding the stationary ...
• 759
19 views

### How to decide $X_n, n \in [0,\infty)$ is a Markov chain and how to compute transition probability matrix?

Three white and three black balls are distributed in two urns in such a way that each contains three balls. We say that the system is in state i, i = 0, 1, 2, 3, if the first urn contains i white ...
• 1,118
29 views

1 vote
50 views

### Generate conditional probability matrix for simulation

I want to generate a categorical random variable conditioned on other variables, for example, generate $X$ from $Y$ by $P(X|Y)$, or generate $X$ from $Y_1,Y_2$ by $P(X|Y_1,Y_2)$. Here, $P(X|Y)$ ...
• 137
16 views

### Reinforcement Learning MDP stochastic policies

I am struggling to get an understanding behind stochastic policy in a MDP I was reading this paper and in section 3.2 they say At each time step $t$, corresponding to extracting sentence number $t$, ...
13 views

### How to define relative value functions for multichain MDPs

I have been studying the average reward MDP, and I found that in most references, the average reward criterion is defined as $\rho(s)=\lim_{N\to\infty} E_\pi[\frac{1}{N}\sum^{N-1}_{t=0}r_\pi(s_t)]$, ...
• 101
23 views

### How to test for equality two finite state markov chains?

I have the transition probabilities of the two markov chains. I think that maybe I could apply the Kullback–Leibler divergence two the stationary distributions of these Markov chains (if they are ...
• 1,400
28 views

### Data Sufficiency for Markov Model

I have a question regarding the Markov model. Suppose I have N number of data in a table and I constructed a simple Markov model with their state transitions as shown below. How do I know if I have &...
569 views

### Can any Models be "Bagged"?

I have been learning about "bagging" (bootstrap aggregation) - supposedly, there are many types of statistical models can be bagged together. For example, CART Decision Trees can be "...
• 6,048
1 vote
24 views

### Memorylessness by way of additional dimensions

This is a somewhat broad question that occurred to me regarding the nature of memorylessness. Namely: Is there utility in considering systems which are themselves not memoryless, but then expanding ...
1 vote
26 views

### Goodness of fit test for a transition density function of a Markov process

Suppose that you have one realization $x = \{x_n\}_{n = 1}^{N}$ of the stochastic process $X = \{X_n\}_{n = 1}^{N}$ with state space $\mathbb{R}$. Assume that the process is Markovian, time-...
• 11
1 vote
12 views

### Coding assignment autograder: Checking if a data stream is generated by a particular Markov process

I teach an introductory programming course in which most assignments ship with a set of tests students can use locally on their machine to validate that their code works as intended. Many years ago we ...
16 views

### continuous time markov chain sample size

Is that the same if I have one patient with one Markov chain of length 100, and 50 patients with 50 Markov chains of length 2? If they are the same, why? Thanks
• 51
45 views

### Balancing "Delayed Entry Bias" and "Survivorship Bias"?

This is a question I have always struggled with - suppose you have medical data on patients over a period of time. This includes information on how long they spent in different states: Admission, ...
• 6,048
49 views

### Can some Survival Models "Dominate" other Survival Models?

I recently heard an interesting interpretation of Survival Models : A "standard" Survival Analysis problem (e.g. where at the end of the study, observations can either be "Censored"...
• 6,048
31 views

### Showing that random process is stationary

Suppose i have $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which positive ...
• 101
59 views

### Why do we need to Define "Valid" State Transitions in a Multi-State Model?

I was watching this video (https://www.youtube.com/watch?v=Wy-WmY6x4tg) and the presenter mentions (@ 8:10) that in a Multi-State Model, the user is required to specify number of "States" ...
• 6,048
119 views

### Markov Chain Monte Carlo with known normalisation

I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
• 141
1 vote
41 views

### Expectation of transition in a Markov process

I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C: \begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\ 0.0265225 & ...
• 123
1 vote
28 views

### How to map a sequence to a transition matrix

I have the following transition matrices, one for Maria and one for Anna: ...
• 391
16 views

### The gamble problem to find limiting stationary distribution

[This question was asked here before, but I did not get satisfactory response] I was trying to solve the following problem: I find out the transition matrix is: P = \begin{bmatrix} 1 & 0 &0 ...
12 views

### How does detailed balance relate to (conditional) expectation?

Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
• 1,560
22 views

### First order PDF of a stochastic process

I've started studying about stochastic processes and I need some help in this question. A random number generator is making numbers by this process: First number (X0) is a sample from Normal Standard ...
1 vote
75 views

### What type of Markov Chain is a random walk of a Knight on a chessboard?

Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
• 129
29 views

### Hidden Markov Model observing sequences

I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
• 143
75 views

### Prove stationary distribution of the CTMC with Cut method

Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$ on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition) ($A$, $A^c$ ) with $A ⊂ S$:...
27 views

### Identification of the transition probability of a time homogeneous MDP with subsampling

I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation. ...
• 137