Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Markov chain position when n=0?

when I look at this chain, Iknow it does not change at n=0. So im looking for clarification on r00=1 which I believe should be 0 and r10 should be 1. Consider the following two-state (0 and 1) ...
user avatar
  • 1
0 votes
0 answers
11 views

Metropolis and Barker Algorithms: Irreducibility

A question from Brémaud, Markov Chains Gibbs Fields, Monte Carlo Simulation and Queues. Exercise 11.5.2. Metropolis and Barker Algorithms: Irreducibility how do you show explicitly that $$p_{i,j}= \...
user avatar
  • 1
3 votes
1 answer
89 views

Markov Chain Monte Carlo with known normalisation

I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
user avatar
  • 131
1 vote
1 answer
15 views

Expectation of transition in a Markov process

I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C: \begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\ 0.0265225 & ...
user avatar
  • 113
1 vote
0 answers
11 views

How to map a sequence to a transition matrix

I have the following transition matrices, one for Maria and one for Anna: ...
user avatar
  • 359
0 votes
0 answers
9 views

The gamble problem to find limiting stationary distribution

[This question was asked here before, but I did not get satisfactory response] I was trying to solve the following problem: I find out the transition matrix is: $$P = \begin{bmatrix} 1 & 0 &0 ...
user avatar
0 votes
0 answers
10 views

How does detailed balance relate to (conditional) expectation?

Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
user avatar
  • 1,458
0 votes
0 answers
14 views

First order PDF of a stochastic process

I've started studying about stochastic processes and I need some help in this question. A random number generator is making numbers by this process: First number (X0) is a sample from Normal Standard ...
user avatar
  • 1
1 vote
1 answer
52 views

What type of Markov Chain is a random walk of a Knight on a chessboard?

Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
user avatar
  • 129
2 votes
0 answers
22 views

Hidden Markov Model observing sequences

I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
user avatar
  • 133
0 votes
0 answers
61 views

Prove stationary distribution of the CTMC with Cut method

Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$ on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition) ($A$, $A^c$ ) with $A ⊂ S$:...
user avatar
  • 1
0 votes
0 answers
24 views

Identification of the transition probability of a time homogeneous MDP with subsampling

I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation. ...
user avatar
0 votes
0 answers
24 views

Why the memoryless Markov Property is desirable

The memoryless Markov property says future predictions only depend on the current status. With longitudinal data, we have all the past data recorded. Why cannot we make use of all the info? Why should ...
user avatar
  • 41
0 votes
0 answers
17 views

How do you consider the negative states i.e. ...-3,-2,-1, when solving for the stationary distribution of this MC?

I have shown that the MC is aperiodic and irreducible, if that helps.
user avatar
  • 1
0 votes
0 answers
22 views

Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability

Question: Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability So essentially given any pair of states a start state and end state in ...
user avatar
  • 129
0 votes
0 answers
11 views

What are good options for deriving probability distributions of transition matrices when data lacks the memoryless property of a Markov chain?

I have a dataset which I initially believed was suitable for running Markov chain simulations, in that there is a finite number of readily identifiable states that all population elements fall into ...
user avatar
0 votes
0 answers
28 views

Confidence Interval for Markov Chain Probability

I have a simple transition model I am trying to use to predict the probability of two states. $$ \begin{bmatrix} p_{1,t+1}\\ p_{2,t+1} \\ \end{bmatrix}= \begin{bmatrix} p_{11} & p_{12} \\ ...
user avatar
  • 151
1 vote
0 answers
7 views

Can we do Deep Reinforcement Learning with Disjoint Action Sets?

I'm defining a construction you can apply to a Markov decision process*, and it involves extending an equivalence relation from the the state space of the MDP to an equivalence relation on the action ...
user avatar
  • 11
0 votes
0 answers
11 views

How to generate states for a Markov-Model through K-Means-Clustering on time-series?

I am reading a paper by Zufferey et al.: https://ieeexplore.ieee.org/document/8442470 On page 2 it says: "In this paper, the definition of the states is based on a K-Means clustering which allows ...
user avatar
0 votes
0 answers
6 views

Testing for Markov Process

Assume you have a box that has a button and a display panel. Every time the button is pressed an integer between 1 and N is displayed. The problem is to test which of two hypothesis is more likely: ...
user avatar
2 votes
1 answer
63 views

Can we rank markov matrices in stochastic order?

I am familiar with the concept of stochastic ordering for two random variables and how we can say if a markov matrix is stochastically monotone. What im interested in is if there is a concept for ...
user avatar
  • 782
2 votes
1 answer
106 views

Markov chain and conditional probabilities

If I define a Markov chain, $$ A\to B\to C \\P(A,B,C)=P(A)P(B|A)P(C|B) $$ Can I derive an expression for $P(C|A)$? I feel like this should be $P(C|B)P(B|A)$, but I am not seeing how to prove it.
user avatar
1 vote
0 answers
14 views

Markov Chains: Do different transition probability matrices imply different stationary distributions?

As in the title. We know that irreducible Markov chains have a invariant distribution which is reached after enough steps (Reference: Billingsley's Probability and Measure, Theorem 8.6). Question: If ...
user avatar
3 votes
1 answer
84 views

Closed form for a markov chain, where transition probabilities depend on $n$?

Let $A_n$ be an event whose success probability $P(A_n)$ depends on the size, $n\in \mathbb{N}$, of a given sample. Let $d$ be a fixed positive integer. For the problem I am working on I can assume a ...
user avatar
0 votes
0 answers
7 views

Some hint for modeling a dynamic with Markov chain?

Consider a kind of game in which there is one card supplier and $n$ player $P=\{p_1,p_2,...p_n\}$ which consumes the cards. suppose the rules of the game are as follows: Card supplier supplies cards ...
user avatar
1 vote
0 answers
15 views

What is the most elegant way to express conditional independence on a line graph?

Consider a Markov graph $$x_1 -x_2-x_3-...-x_t$$ In such a graphical model, we have the conditional independence property $x_{s-1} \perp x_{s+1:t} | x_s \;\forall\; x=2,...,t-1$ and $x_{1:s-1} \perp ...
user avatar
  • 419
0 votes
0 answers
12 views

Is a non markov process same as higher ordered markov process?

In the context of Markov property, I understand processes can be of 2 types: Markov process and Non-markov process. The former says "probability of transition to next state depends only on ...
user avatar
0 votes
0 answers
82 views

Python: Markov switching model out of sample forecasts

Is there a way to obtain out of sample forecasts for Markov switching models estimated via statsmodels (or any other package)? https://www.statsmodels.org/dev/examples/notebooks/generated/...
user avatar
  • 123
1 vote
0 answers
28 views

Ergodic theorem for Markov chains

I am reading Robert and Casella (2004) on Markov Chain Monte Carlo methods and, in particular, Section 6.7. This contains the ergodic theorem, which is stated as follows, where $S_n(f)$ denotes a ...
user avatar
  • 388
0 votes
0 answers
10 views

Uncertain on how to model an updating process

How should I go about finding an answer to the following question? I have a coin that lands heads with some unknown probability $P$, which I represent with a beta prior distribution $beta(a,b)$, each ...
user avatar
0 votes
0 answers
14 views

Filtering and smoothing probability in Markov switching framework

I have been working on Markov chains and have been studying Markov Regime switching models. I have been confused over the term used in Markov switching models called Filtered and Smoothed ...
user avatar
  • 1
0 votes
0 answers
7 views

Do time intervals have to be consistant for a Markov chain?

My set up is as follows. I have about 50 groups of people who move about 5 states. One of these states is essentially "not-doing anything". I have their states recorded at the time-scale of ...
user avatar
2 votes
1 answer
83 views

Stationary distribution of an infinite Markov chain

The exercise is asking for the stationary distribution, the estimated time to get from state $0$ to state $4,$ and to conclude if the chain is time-reversible. So I have the following transition ...
user avatar
  • 21
1 vote
0 answers
41 views

How were the probabilities calculated [closed]

I'm trying to calculate the probabilities in the paper 1 but I'm getting different results. In the paper, the forecasted smoothed probabilities are calculated as follows: $$\mathbb{P}(S_{t+1} = i|\...
user avatar
1 vote
0 answers
38 views

Real-time sequence classification with Markov Chains vs HMM vs CRF

I see that Markov Chains are useful for providing the conditional probabilities for each individual symbol of the test sequence. So this really gives an incremental overview on how the sequence is ...
user avatar
0 votes
0 answers
17 views

How to go on about building a Markov regime-switching based early warning system?

I would like to build a Markov regime-switching based early warning system. From the several papers I've skimmed through, [1][2][3][4] they go on about estimating a Markov regime-switching model as a ...
user avatar
0 votes
1 answer
43 views

Proposed transition matrix for MCMC in two-state Markov Chain

Suppose we would like to model the weather (either sunny $S$ or cloudy $C$) using a two-state Markov Chain, given a set of data collected from 10000 days: $$CCCSSSSSSCCCSSSSSSCCCC...$$ We can use the ...
user avatar
  • 101
0 votes
0 answers
19 views

How do HHMM (Hierarchical Hidden Markov Models) work and where to learn more?

I recently came across something called Hierarchical Hidden Markov Models. I am familiar with HMMs, but not HHMMs. I have two questions. I can't fully understand the procedure after reading here: ...
user avatar
0 votes
1 answer
37 views

What are the transition functions for RNNs

From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model. Thus, just as probabilistic ...
user avatar
1 vote
0 answers
24 views

How to create/design a Hidden Markov Model?

I have a rough conceptual understanding of what Hidden Markov Models do. What I don't understand is how to really create/train one. Let me outline what I'm working on, and then I'll give more specific ...
user avatar
0 votes
0 answers
51 views

R CODE Transition Matrix having zeroes for some transitions

I have been trying to create a Transition Matrix using the data from 2000 entities over 40 observations (Years). I have ranked the data into percentiles, for example the highest value entity in a ...
user avatar
0 votes
0 answers
23 views

Estimating transition probabilities of a Markov chain with bayesian approach

I am trying to model heteroskedasticity in time series data,and the volatility $\left(\sigma_{t}\right)$ is taken as a Markov chain with two values $\sigma_{h}>\sigma_{l}>0$ with transition ...
user avatar
0 votes
0 answers
14 views

Formulation of ARIMA(1,1,1) as markov process by extended state space [duplicate]

my question is: How can I formulate ARIMA(1,1,1) as Markov Process by extended state space? i.e. if $$s_t = \mu + \sum_{i=1}^p\beta_i s_{t-i} + \varepsilon_t$$ Define$$S_t = (s_t,\ldots,s_{t-p+1})$$ ...
user avatar
1 vote
1 answer
44 views

What is a mixture of RNNs?

I am reading papers on different types of classification and prediction methods and keep coming across "Mixture of Recurrent Neural Networks" and "Mixture of Markov Chain Models". ...
user avatar
2 votes
1 answer
149 views

interpretation of Markov transition field

Markov transition fields come from this article: https://www.researchgate.net/publication/...
user avatar
  • 71
4 votes
2 answers
442 views

Prove that a process is memoryless (simple example)

Given the following stochastic process \begin{equation} x_t = \frac{u_t}{\sum_{s=1}^{t-1} u_s} \end{equation} where $u_t \overset{i.i.d.}{\sim} \mathcal{N}(0,\sigma^2)$, $\sigma^2<\infty$, and ...
user avatar
0 votes
0 answers
28 views

Bayes' Rule for second order Markov Chain

I want build two second order Markov Chains and compute the conditional probability naturally through Bayes' rule. Assuming for a sequence of transactions, say $T_i=(t_1,\dotsc,t_i)$, I want to find ...
user avatar
1 vote
0 answers
28 views

A Markov Regime-Switching GARCH with Time-Varying Transition Matrix Package in R

Does anyone know if there exists any Markov regime-switching GARCH with time-varying transition matrix package or tutorial in R? I know of the "MSGARCH" package by D. Ardia et al. but the ...
user avatar
1 vote
0 answers
50 views

How do I get the stationary distribution of a Markov chain matrix from SVD?

I have a matrix that represents a Markov chain. ...
user avatar
  • 1,298
1 vote
0 answers
24 views

How do Hidden Markov models classify sequential data?

How exactly do HMMs classify sequential data? I understand that this is a generative model, which models the joint probability distribution and provides us with the conditional probability of ...
user avatar

1
2 3 4 5
25