# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,235 questions
Filter by
Sorted by
Tagged with
12 views

### Markov chain position when n=0?

when I look at this chain, Iknow it does not change at n=0. So im looking for clarification on r00=1 which I believe should be 0 and r10 should be 1. Consider the following two-state (0 and 1) ...
11 views

10 views

### How does detailed balance relate to (conditional) expectation?

Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
• 1,458
14 views

### First order PDF of a stochastic process

I've started studying about stochastic processes and I need some help in this question. A random number generator is making numbers by this process: First number (X0) is a sample from Normal Standard ...
1 vote
52 views

### What type of Markov Chain is a random walk of a Knight on a chessboard?

Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
• 129
22 views

### Hidden Markov Model observing sequences

I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
• 133
61 views

### Prove stationary distribution of the CTMC with Cut method

Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$ on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition) ($A$, $A^c$ ) with $A ⊂ S$:...
24 views

### Identification of the transition probability of a time homogeneous MDP with subsampling

I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation. ...
24 views

### Why the memoryless Markov Property is desirable

The memoryless Markov property says future predictions only depend on the current status. With longitudinal data, we have all the past data recorded. Why cannot we make use of all the info? Why should ...
• 41
17 views

### How do you consider the negative states i.e. ...-3,-2,-1, when solving for the stationary distribution of this MC?

I have shown that the MC is aperiodic and irreducible, if that helps.
• 1
22 views

### Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability

Question: Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability So essentially given any pair of states a start state and end state in ...
• 129
11 views

### What are good options for deriving probability distributions of transition matrices when data lacks the memoryless property of a Markov chain?

I have a dataset which I initially believed was suitable for running Markov chain simulations, in that there is a finite number of readily identifiable states that all population elements fall into ...
28 views

1 vote
38 views

### Real-time sequence classification with Markov Chains vs HMM vs CRF

I see that Markov Chains are useful for providing the conditional probabilities for each individual symbol of the test sequence. So this really gives an incremental overview on how the sequence is ...
• 135
17 views

### How to go on about building a Markov regime-switching based early warning system?

I would like to build a Markov regime-switching based early warning system. From the several papers I've skimmed through, [1][2][3][4] they go on about estimating a Markov regime-switching model as a ...
• 638
43 views

### Proposed transition matrix for MCMC in two-state Markov Chain

Suppose we would like to model the weather (either sunny $S$ or cloudy $C$) using a two-state Markov Chain, given a set of data collected from 10000 days: $$CCCSSSSSSCCCSSSSSSCCCC...$$ We can use the ...
• 101
19 views

### How do HHMM (Hierarchical Hidden Markov Models) work and where to learn more?

I recently came across something called Hierarchical Hidden Markov Models. I am familiar with HMMs, but not HHMMs. I have two questions. I can't fully understand the procedure after reading here: ...
• 1,082
37 views

### What are the transition functions for RNNs

From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model. Thus, just as probabilistic ...
• 135
1 vote
24 views

### How to create/design a Hidden Markov Model?

I have a rough conceptual understanding of what Hidden Markov Models do. What I don't understand is how to really create/train one. Let me outline what I'm working on, and then I'll give more specific ...
• 1,082
51 views

### R CODE Transition Matrix having zeroes for some transitions

I have been trying to create a Transition Matrix using the data from 2000 entities over 40 observations (Years). I have ranked the data into percentiles, for example the highest value entity in a ...
23 views

### Estimating transition probabilities of a Markov chain with bayesian approach

I am trying to model heteroskedasticity in time series data,and the volatility $\left(\sigma_{t}\right)$ is taken as a Markov chain with two values $\sigma_{h}>\sigma_{l}>0$ with transition ...
14 views

### Formulation of ARIMA(1,1,1) as markov process by extended state space [duplicate]

my question is: How can I formulate ARIMA(1,1,1) as Markov Process by extended state space? i.e. if $$s_t = \mu + \sum_{i=1}^p\beta_i s_{t-i} + \varepsilon_t$$ Define$$S_t = (s_t,\ldots,s_{t-p+1})$$ ...
1 vote
44 views

### What is a mixture of RNNs?

I am reading papers on different types of classification and prediction methods and keep coming across "Mixture of Recurrent Neural Networks" and "Mixture of Markov Chain Models". ...
• 135
149 views

• 71
442 views

### Prove that a process is memoryless (simple example)

Given the following stochastic process $$x_t = \frac{u_t}{\sum_{s=1}^{t-1} u_s}$$ where $u_t \overset{i.i.d.}{\sim} \mathcal{N}(0,\sigma^2)$, $\sigma^2<\infty$, and ...
28 views

### Bayes' Rule for second order Markov Chain

I want build two second order Markov Chains and compute the conditional probability naturally through Bayes' rule. Assuming for a sequence of transactions, say $T_i=(t_1,\dotsc,t_i)$, I want to find ...
• 135
1 vote
28 views

### A Markov Regime-Switching GARCH with Time-Varying Transition Matrix Package in R

Does anyone know if there exists any Markov regime-switching GARCH with time-varying transition matrix package or tutorial in R? I know of the "MSGARCH" package by D. Ardia et al. but the ...
• 638
1 vote
50 views

### How do I get the stationary distribution of a Markov chain matrix from SVD?

I have a matrix that represents a Markov chain. ...
• 1,298
1 vote