Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

How to define relative value functions for multichain MDPs

I have been studying the average reward MDP, and I found that in most references, the average reward criterion is defined as $\rho(s)=\lim_{N\to\infty} E_\pi[\frac{1}{N}\sum^{N-1}_{t=0}r_\pi(s_t)]$, ...
user avatar
0 votes
0 answers
17 views

Markov chain of independent variables [closed]

Let $X$ and $Y$ be discrete and independent. Consider $Z = \{ X, Y \}$ and a Markov chain $Z \rightarrow Z_1 \rightarrow \dots \rightarrow Z_N$. Can we write $Z_i = \{ X_i, Y_i \}$, with $X_i = f(X)$ ...
user avatar
  • 617
0 votes
0 answers
10 views

How to test for equality two finite state markov chains?

I have the transition probabilities of the two markov chains. I think that maybe I could apply the Kullback–Leibler divergence two the stationary distributions of these Markov chains (if they are ...
user avatar
0 votes
0 answers
12 views

Data Sufficiency for Markov Model

I have a question regarding the Markov model. Suppose I have N number of data in a table and I constructed a simple Markov model with their state transitions as shown below. How do I know if I have &...
user avatar
6 votes
2 answers
547 views

Can any Models be "Bagged"?

I have been learning about "bagging" (bootstrap aggregation) - supposedly, there are many types of statistical models can be bagged together. For example, CART Decision Trees can be "...
user avatar
  • 5,892
1 vote
0 answers
18 views

Memorylessness by way of additional dimensions

This is a somewhat broad question that occurred to me regarding the nature of memorylessness. Namely: Is there utility in considering systems which are themselves not memoryless, but then expanding ...
user avatar
1 vote
1 answer
19 views

Goodness of fit test for a transition density function of a Markov process

Suppose that you have one realization $x = \{x_n\}_{n = 1}^{N}$ of the stochastic process $X = \{X_n\}_{n = 1}^{N}$ with state space $\mathbb{R}$. Assume that the process is Markovian, time-...
user avatar
  • 11
1 vote
0 answers
10 views

Coding assignment autograder: Checking if a data stream is generated by a particular Markov process

I teach an introductory programming course in which most assignments ship with a set of tests students can use locally on their machine to validate that their code works as intended. Many years ago we ...
user avatar
0 votes
0 answers
11 views

continuous time markov chain sample size

Is that the same if I have one patient with one Markov chain of length 100, and 50 patients with 50 Markov chains of length 2? If they are the same, why? Thanks
user avatar
  • 41
0 votes
1 answer
36 views

Balancing "Delayed Entry Bias" and "Survivorship Bias"?

This is a question I have always struggled with - suppose you have medical data on patients over a period of time. This includes information on how long they spent in different states: Admission, ...
user avatar
  • 5,892
0 votes
1 answer
40 views

Can some Survival Models "Dominate" other Survival Models?

I recently heard an interesting interpretation of Survival Models : A "standard" Survival Analysis problem (e.g. where at the end of the study, observations can either be "Censored"...
user avatar
  • 5,892
0 votes
0 answers
29 views

Showing that random process is stationary

Suppose i have $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which positive ...
user avatar
2 votes
1 answer
53 views

Why do we need to Define "Valid" State Transitions in a Multi-State Model?

I was watching this video (https://www.youtube.com/watch?v=Wy-WmY6x4tg) and the presenter mentions (@ 8:10) that in a Multi-State Model, the user is required to specify number of "States" ...
user avatar
  • 5,892
4 votes
1 answer
108 views

Markov Chain Monte Carlo with known normalisation

I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
user avatar
  • 141
1 vote
1 answer
23 views

Expectation of transition in a Markov process

I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C: \begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\ 0.0265225 & ...
user avatar
  • 113
1 vote
0 answers
15 views

How to map a sequence to a transition matrix

I have the following transition matrices, one for Maria and one for Anna: ...
user avatar
  • 391
0 votes
0 answers
11 views

The gamble problem to find limiting stationary distribution

[This question was asked here before, but I did not get satisfactory response] I was trying to solve the following problem: I find out the transition matrix is: $$P = \begin{bmatrix} 1 & 0 &0 ...
user avatar
0 votes
0 answers
11 views

How does detailed balance relate to (conditional) expectation?

Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
user avatar
  • 1,488
0 votes
0 answers
16 views

First order PDF of a stochastic process

I've started studying about stochastic processes and I need some help in this question. A random number generator is making numbers by this process: First number (X0) is a sample from Normal Standard ...
user avatar
  • 1
1 vote
1 answer
65 views

What type of Markov Chain is a random walk of a Knight on a chessboard?

Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
user avatar
  • 129
2 votes
0 answers
28 views

Hidden Markov Model observing sequences

I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
user avatar
  • 133
0 votes
0 answers
64 views

Prove stationary distribution of the CTMC with Cut method

Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$ on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition) ($A$, $A^c$ ) with $A ⊂ S$:...
user avatar
  • 1
0 votes
0 answers
25 views

Identification of the transition probability of a time homogeneous MDP with subsampling

I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation. ...
user avatar
0 votes
0 answers
26 views

Why the memoryless Markov Property is desirable

The memoryless Markov property says future predictions only depend on the current status. With longitudinal data, we have all the past data recorded. Why cannot we make use of all the info? Why should ...
user avatar
  • 41
0 votes
0 answers
20 views

How do you consider the negative states i.e. ...-3,-2,-1, when solving for the stationary distribution of this MC?

I have shown that the MC is aperiodic and irreducible, if that helps.
user avatar
  • 1
0 votes
0 answers
28 views

Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability

Question: Given an irreducible Markov Chain. Prove you can reach any pair of states in $N$ steps with greater than 0 probability So essentially given any pair of states a start state and end state in ...
user avatar
  • 129
0 votes
0 answers
11 views

What are good options for deriving probability distributions of transition matrices when data lacks the memoryless property of a Markov chain?

I have a dataset which I initially believed was suitable for running Markov chain simulations, in that there is a finite number of readily identifiable states that all population elements fall into ...
user avatar
0 votes
0 answers
41 views

Confidence Interval for Markov Chain Probability

I have a simple transition model I am trying to use to predict the probability of two states. $$ \begin{bmatrix} p_{1,t+1}\\ p_{2,t+1} \\ \end{bmatrix}= \begin{bmatrix} p_{11} & p_{12} \\ ...
user avatar
  • 151
1 vote
0 answers
8 views

Can we do Deep Reinforcement Learning with Disjoint Action Sets?

I'm defining a construction you can apply to a Markov decision process*, and it involves extending an equivalence relation from the the state space of the MDP to an equivalence relation on the action ...
user avatar
  • 11
0 votes
0 answers
11 views

How to generate states for a Markov-Model through K-Means-Clustering on time-series?

I am reading a paper by Zufferey et al.: https://ieeexplore.ieee.org/document/8442470 On page 2 it says: "In this paper, the definition of the states is based on a K-Means clustering which allows ...
user avatar
0 votes
0 answers
10 views

Testing for Markov Process

Assume you have a box that has a button and a display panel. Every time the button is pressed an integer between 1 and N is displayed. The problem is to test which of two hypothesis is more likely: ...
user avatar
2 votes
1 answer
64 views

Can we rank markov matrices in stochastic order?

I am familiar with the concept of stochastic ordering for two random variables and how we can say if a markov matrix is stochastically monotone. What im interested in is if there is a concept for ...
user avatar
  • 792
2 votes
1 answer
126 views

Markov chain and conditional probabilities

If I define a Markov chain, $$ A\to B\to C \\P(A,B,C)=P(A)P(B|A)P(C|B) $$ Can I derive an expression for $P(C|A)$? I feel like this should be $P(C|B)P(B|A)$, but I am not seeing how to prove it.
user avatar
1 vote
0 answers
18 views

Markov Chains: Do different transition probability matrices imply different stationary distributions?

As in the title. We know that irreducible Markov chains have a invariant distribution which is reached after enough steps (Reference: Billingsley's Probability and Measure, Theorem 8.6). Question: If ...
user avatar
4 votes
1 answer
95 views

Closed form for a markov chain, where transition probabilities depend on $n$?

Let $A_n$ be an event whose success probability $P(A_n)$ depends on the size, $n\in \mathbb{N}$, of a given sample. Let $d$ be a fixed positive integer. For the problem I am working on I can assume a ...
user avatar
0 votes
0 answers
7 views

Some hint for modeling a dynamic with Markov chain?

Consider a kind of game in which there is one card supplier and $n$ player $P=\{p_1,p_2,...p_n\}$ which consumes the cards. suppose the rules of the game are as follows: Card supplier supplies cards ...
user avatar
1 vote
0 answers
15 views

What is the most elegant way to express conditional independence on a line graph?

Consider a Markov graph $$x_1 -x_2-x_3-...-x_t$$ In such a graphical model, we have the conditional independence property $x_{s-1} \perp x_{s+1:t} | x_s \;\forall\; x=2,...,t-1$ and $x_{1:s-1} \perp ...
user avatar
  • 421
0 votes
0 answers
16 views

Is a non markov process same as higher ordered markov process?

In the context of Markov property, I understand processes can be of 2 types: Markov process and Non-markov process. The former says "probability of transition to next state depends only on ...
user avatar
0 votes
0 answers
156 views

Python: Markov switching model out of sample forecasts

Is there a way to obtain out of sample forecasts for Markov switching models estimated via statsmodels (or any other package)? https://www.statsmodels.org/dev/examples/notebooks/generated/...
user avatar
  • 123
1 vote
0 answers
31 views

Ergodic theorem for Markov chains

I am reading Robert and Casella (2004) on Markov Chain Monte Carlo methods and, in particular, Section 6.7. This contains the ergodic theorem, which is stated as follows, where $S_n(f)$ denotes a ...
user avatar
  • 388
0 votes
0 answers
12 views

Uncertain on how to model an updating process

How should I go about finding an answer to the following question? I have a coin that lands heads with some unknown probability $P$, which I represent with a beta prior distribution $beta(a,b)$, each ...
user avatar
0 votes
0 answers
29 views

Filtering and smoothing probability in Markov switching framework

I have been working on Markov chains and have been studying Markov Regime switching models. I have been confused over the term used in Markov switching models called Filtered and Smoothed ...
user avatar
  • 1
0 votes
0 answers
7 views

Do time intervals have to be consistant for a Markov chain?

My set up is as follows. I have about 50 groups of people who move about 5 states. One of these states is essentially "not-doing anything". I have their states recorded at the time-scale of ...
user avatar
2 votes
1 answer
108 views

Stationary distribution of an infinite Markov chain

The exercise is asking for the stationary distribution, the estimated time to get from state $0$ to state $4,$ and to conclude if the chain is time-reversible. So I have the following transition ...
user avatar
  • 21
1 vote
0 answers
42 views

How were the probabilities calculated [closed]

I'm trying to calculate the probabilities in the paper 1 but I'm getting different results. In the paper, the forecasted smoothed probabilities are calculated as follows: $$\mathbb{P}(S_{t+1} = i|\...
user avatar
1 vote
0 answers
48 views

Real-time sequence classification with Markov Chains vs HMM vs CRF

I see that Markov Chains are useful for providing the conditional probabilities for each individual symbol of the test sequence. So this really gives an incremental overview on how the sequence is ...
user avatar
0 votes
0 answers
21 views

How to go on about building a Markov regime-switching based early warning system?

I would like to build a Markov regime-switching based early warning system. From the several papers I've skimmed through, [1][2][3][4] they go on about estimating a Markov regime-switching model as a ...
user avatar
0 votes
1 answer
53 views

Proposed transition matrix for MCMC in two-state Markov Chain

Suppose we would like to model the weather (either sunny $S$ or cloudy $C$) using a two-state Markov Chain, given a set of data collected from 10000 days: $$CCCSSSSSSCCCSSSSSSCCCC...$$ We can use the ...
user avatar
  • 101
0 votes
0 answers
20 views

How do HHMM (Hierarchical Hidden Markov Models) work and where to learn more?

I recently came across something called Hierarchical Hidden Markov Models. I am familiar with HMMs, but not HHMMs. I have two questions. I can't fully understand the procedure after reading here: ...
user avatar
0 votes
1 answer
48 views

What are the transition functions for RNNs

From what I understand, the hidden states of RNNs are equivalent to the deterministic probability distribution over hidden states in for example a Hidden Markov Model. Thus, just as probabilistic ...
user avatar

1
2 3 4 5
25