Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,235
questions
0
votes
0
answers
12
views
Markov chain position when n=0?
when I look at this chain, Iknow it does not change at n=0. So im looking for clarification on r00=1 which I believe should be 0 and r10 should be 1.
Consider the following two-state (0 and 1) ...
- 1
0
votes
0
answers
11
views
Metropolis and Barker Algorithms: Irreducibility
A question from Brémaud, Markov Chains Gibbs Fields, Monte Carlo Simulation and Queues.
Exercise 11.5.2. Metropolis and Barker Algorithms: Irreducibility
how do you show explicitly that
$$p_{i,j}= \...
- 1
3
votes
1
answer
89
views
Markov Chain Monte Carlo with known normalisation
I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
- 131
1
vote
1
answer
15
views
Expectation of transition in a Markov process
I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C:
\begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\
0.0265225 & ...
- 113
1
vote
0
answers
11
views
How to map a sequence to a transition matrix
I have the following transition matrices, one for Maria and one for Anna:
...
- 359
0
votes
0
answers
9
views
The gamble problem to find limiting stationary distribution
[This question was asked here before, but I did not get satisfactory response]
I was trying to solve the following problem:
I find out the transition matrix is:
$$P = \begin{bmatrix}
1 & 0 &0 ...
- 243
0
votes
0
answers
10
views
How does detailed balance relate to (conditional) expectation?
Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
- 1,458
0
votes
0
answers
14
views
First order PDF of a stochastic process
I've started studying about stochastic processes and I need some help in this question.
A random number generator is making numbers by this process:
First number (X0) is a sample from Normal Standard ...
- 1
1
vote
1
answer
52
views
What type of Markov Chain is a random walk of a Knight on a chessboard?
Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
- 129
2
votes
0
answers
22
views
Hidden Markov Model observing sequences
I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
- 133
0
votes
0
answers
61
views
Prove stationary distribution of the CTMC with Cut method
Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$
on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition)
($A$, $A^c$
) with $A ⊂ S$:...
- 1
0
votes
0
answers
24
views
Identification of the transition probability of a time homogeneous MDP with subsampling
I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation.
...
- 57
0
votes
0
answers
24
views
Why the memoryless Markov Property is desirable
The memoryless Markov property says future predictions only depend on the current status.
With longitudinal data, we have all the past data recorded. Why cannot we make use of all the info? Why should ...
- 41
0
votes
0
answers
17
views
How do you consider the negative states i.e. ...-3,-2,-1, when solving for the stationary distribution of this MC?
I have shown that the MC is aperiodic and irreducible, if that helps.
- 1
0
votes
0
answers
22
views
Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability
Question: Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability
So essentially given any pair of states a start state and end state in ...
- 129
0
votes
0
answers
11
views
What are good options for deriving probability distributions of transition matrices when data lacks the memoryless property of a Markov chain?
I have a dataset which I initially believed was suitable for running Markov chain simulations, in that there is a finite number of readily identifiable states that all population elements fall into ...
0
votes
0
answers
28
views
Confidence Interval for Markov Chain Probability
I have a simple transition model I am trying to use to predict the probability of two states.
$$
\begin{bmatrix}
p_{1,t+1}\\
p_{2,t+1} \\
\end{bmatrix}=
\begin{bmatrix}
p_{11} & p_{12} \\
...
- 151
1
vote
0
answers
7
views
Can we do Deep Reinforcement Learning with Disjoint Action Sets?
I'm defining a construction you can apply to a Markov decision process*, and it involves extending an equivalence relation from the the state space of the MDP to an equivalence relation on the action ...
- 11
0
votes
0
answers
11
views
How to generate states for a Markov-Model through K-Means-Clustering on time-series?
I am reading a paper by Zufferey et al.: https://ieeexplore.ieee.org/document/8442470
On page 2 it says:
"In this paper, the definition of the states is based on a K-Means
clustering which allows ...
- 35
0
votes
0
answers
6
views
Testing for Markov Process
Assume you have a box that has a button and a display panel. Every time the button is pressed an integer between 1 and N is displayed. The problem is to test which of two hypothesis is more likely:
...
- 1
2
votes
1
answer
63
views
Can we rank markov matrices in stochastic order?
I am familiar with the concept of stochastic ordering for two random variables and how we can say if a markov matrix is stochastically monotone. What im interested in is if there is a concept for ...
- 782
2
votes
1
answer
106
views
Markov chain and conditional probabilities
If I define a Markov chain,
$$
A\to B\to C
\\P(A,B,C)=P(A)P(B|A)P(C|B)
$$
Can I derive an expression for $P(C|A)$?
I feel like this should be $P(C|B)P(B|A)$, but I am not seeing how to prove it.
- 133
1
vote
0
answers
14
views
Markov Chains: Do different transition probability matrices imply different stationary distributions?
As in the title.
We know that irreducible Markov chains have a invariant distribution which is reached after enough steps (Reference: Billingsley's Probability and Measure, Theorem 8.6).
Question: If ...
- 11
3
votes
1
answer
84
views
Closed form for a markov chain, where transition probabilities depend on $n$?
Let $A_n$ be an event whose success probability $P(A_n)$ depends on the size, $n\in \mathbb{N}$, of a given sample.
Let $d$ be a fixed positive integer.
For the problem I am working on I can assume a ...
- 732
0
votes
0
answers
7
views
Some hint for modeling a dynamic with Markov chain?
Consider a kind of game in which there is one card supplier and $n$ player $P=\{p_1,p_2,...p_n\}$ which consumes the cards. suppose the rules of the game are as follows:
Card supplier supplies cards ...
- 123
1
vote
0
answers
15
views
What is the most elegant way to express conditional independence on a line graph?
Consider a Markov graph
$$x_1 -x_2-x_3-...-x_t$$
In such a graphical model, we have the conditional independence property $x_{s-1} \perp x_{s+1:t} | x_s \;\forall\; x=2,...,t-1$ and $x_{1:s-1} \perp ...
- 419
0
votes
0
answers
12
views
Is a non markov process same as higher ordered markov process?
In the context of Markov property, I understand processes can be of 2 types: Markov process and Non-markov process. The former says "probability of transition to next state depends only on ...
- 101
0
votes
0
answers
82
views
Python: Markov switching model out of sample forecasts
Is there a way to obtain out of sample forecasts for Markov switching models estimated via statsmodels (or any other package)?
https://www.statsmodels.org/dev/examples/notebooks/generated/...
- 123
1
vote
0
answers
28
views
Ergodic theorem for Markov chains
I am reading Robert and Casella (2004) on Markov Chain Monte Carlo methods and, in particular, Section 6.7. This contains the ergodic theorem, which is stated as follows, where $S_n(f)$ denotes a ...
- 388
0
votes
0
answers
10
views
Uncertain on how to model an updating process
How should I go about finding an answer to the following question?
I have a coin that lands heads with some unknown probability $P$, which I represent with a beta prior distribution $beta(a,b)$, each ...
0
votes
0
answers
14
views
Filtering and smoothing probability in Markov switching framework
I have been working on Markov chains and have been studying Markov Regime switching models. I have been confused over the term used in Markov switching models called Filtered and Smoothed ...
- 1
0
votes
0
answers
7
views
Do time intervals have to be consistant for a Markov chain?
My set up is as follows. I have about 50 groups of people who move about 5 states. One of these states is essentially "not-doing anything". I have their states recorded at the time-scale of ...
2
votes
1
answer
83
views
Stationary distribution of an infinite Markov chain
The exercise is asking for the stationary distribution, the estimated time to get from state $0$ to state $4,$ and to conclude if the chain is time-reversible.
So I have the following transition ...
- 21
1
vote
0
answers
41
views
How were the probabilities calculated [closed]
I'm trying to calculate the probabilities in the paper 1 but I'm getting different results. In the paper, the forecasted smoothed probabilities are calculated as follows:
$$\mathbb{P}(S_{t+1} = i|\...
1
vote
0
answers
38
views
Real-time sequence classification with Markov Chains vs HMM vs CRF
I see that Markov Chains are useful for providing the conditional probabilities for each individual symbol of the test sequence. So this really gives an incremental overview on how the sequence is ...
- 135
0
votes
0
answers
17
views
How to go on about building a Markov regime-switching based early warning system?
I would like to build a Markov regime-switching based early warning system. From the several papers I've skimmed through, [1][2][3][4] they go on about estimating a Markov regime-switching model as a ...
- 638
0
votes
1
answer
43
views
Proposed transition matrix for MCMC in two-state Markov Chain
Suppose we would like to model the weather (either sunny $S$ or cloudy $C$) using a two-state Markov Chain, given a set of data collected from 10000 days:
$$CCCSSSSSSCCCSSSSSSCCCC...$$
We can use the ...
- 101
0
votes
0
answers
19
views
How do HHMM (Hierarchical Hidden Markov Models) work and where to learn more?
I recently came across something called Hierarchical Hidden Markov Models. I am familiar with HMMs, but not HHMMs.
I have two questions.
I can't fully understand the procedure after reading here: ...
- 1,082
0
votes
1
answer
37
views
What are the transition functions for RNNs
From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model.
Thus, just as probabilistic ...
- 135
1
vote
0
answers
24
views
How to create/design a Hidden Markov Model?
I have a rough conceptual understanding of what Hidden Markov Models do.
What I don't understand is how to really create/train one. Let me outline what I'm working on, and then I'll give more specific ...
- 1,082
0
votes
0
answers
51
views
R CODE Transition Matrix having zeroes for some transitions
I have been trying to create a Transition Matrix using the data from 2000 entities over 40 observations (Years). I have ranked the data into percentiles, for example the highest value entity in a ...
0
votes
0
answers
23
views
Estimating transition probabilities of a Markov chain with bayesian approach
I am trying to model heteroskedasticity in time series data,and the volatility $\left(\sigma_{t}\right)$ is taken as a Markov chain with two values $\sigma_{h}>\sigma_{l}>0$ with transition ...
- 3
0
votes
0
answers
14
views
Formulation of ARIMA(1,1,1) as markov process by extended state space [duplicate]
my question is: How can I formulate ARIMA(1,1,1) as Markov Process by extended state space? i.e. if $$s_t = \mu + \sum_{i=1}^p\beta_i s_{t-i} + \varepsilon_t$$ Define$$S_t = (s_t,\ldots,s_{t-p+1})$$ ...
- 3
1
vote
1
answer
44
views
What is a mixture of RNNs?
I am reading papers on different types of classification and prediction methods and keep coming across "Mixture of Recurrent Neural Networks" and "Mixture of Markov Chain Models".
...
- 135
2
votes
1
answer
149
views
interpretation of Markov transition field
Markov transition fields come from this article:
https://www.researchgate.net/publication/...
- 71
4
votes
2
answers
442
views
Prove that a process is memoryless (simple example)
Given the following stochastic process
\begin{equation}
x_t = \frac{u_t}{\sum_{s=1}^{t-1} u_s}
\end{equation}
where $u_t \overset{i.i.d.}{\sim} \mathcal{N}(0,\sigma^2)$, $\sigma^2<\infty$, and ...
0
votes
0
answers
28
views
Bayes' Rule for second order Markov Chain
I want build two second order Markov Chains and compute the conditional probability naturally through Bayes' rule. Assuming for a sequence of transactions, say $T_i=(t_1,\dotsc,t_i)$, I want to find ...
- 135
1
vote
0
answers
28
views
A Markov Regime-Switching GARCH with Time-Varying Transition Matrix Package in R
Does anyone know if there exists any Markov regime-switching GARCH with time-varying transition matrix package or tutorial in R? I know of the "MSGARCH" package by D. Ardia et al. but the ...
- 638
1
vote
0
answers
50
views
How do I get the stationary distribution of a Markov chain matrix from SVD?
I have a matrix that represents a Markov chain.
...
- 1,298
1
vote
0
answers
24
views
How do Hidden Markov models classify sequential data?
How exactly do HMMs classify sequential data? I understand that this is a generative model, which models the joint probability distribution and provides us with the conditional probability of ...
- 135