# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,244 questions
Filter by
Sorted by
Tagged with
8 views

### How to define relative value functions for multichain MDPs

I have been studying the average reward MDP, and I found that in most references, the average reward criterion is defined as $\rho(s)=\lim_{N\to\infty} E_\pi[\frac{1}{N}\sum^{N-1}_{t=0}r_\pi(s_t)]$, ...
• 101
17 views

### Markov chain of independent variables [closed]

Let $X$ and $Y$ be discrete and independent. Consider $Z = \{ X, Y \}$ and a Markov chain $Z \rightarrow Z_1 \rightarrow \dots \rightarrow Z_N$. Can we write $Z_i = \{ X_i, Y_i \}$, with $X_i = f(X)$ ...
• 617
10 views

### How to test for equality two finite state markov chains?

I have the transition probabilities of the two markov chains. I think that maybe I could apply the Kullback–Leibler divergence two the stationary distributions of these Markov chains (if they are ...
• 1,400
12 views

### Data Sufficiency for Markov Model

I have a question regarding the Markov model. Suppose I have N number of data in a table and I constructed a simple Markov model with their state transitions as shown below. How do I know if I have &...
547 views

### Can any Models be "Bagged"?

I have been learning about "bagging" (bootstrap aggregation) - supposedly, there are many types of statistical models can be bagged together. For example, CART Decision Trees can be "...
• 5,892
1 vote
18 views

### Memorylessness by way of additional dimensions

This is a somewhat broad question that occurred to me regarding the nature of memorylessness. Namely: Is there utility in considering systems which are themselves not memoryless, but then expanding ...
1 vote
19 views

### Goodness of fit test for a transition density function of a Markov process

Suppose that you have one realization $x = \{x_n\}_{n = 1}^{N}$ of the stochastic process $X = \{X_n\}_{n = 1}^{N}$ with state space $\mathbb{R}$. Assume that the process is Markovian, time-...
• 11
1 vote
10 views

### Coding assignment autograder: Checking if a data stream is generated by a particular Markov process

I teach an introductory programming course in which most assignments ship with a set of tests students can use locally on their machine to validate that their code works as intended. Many years ago we ...
11 views

### continuous time markov chain sample size

Is that the same if I have one patient with one Markov chain of length 100, and 50 patients with 50 Markov chains of length 2? If they are the same, why? Thanks
• 41
36 views

### Balancing "Delayed Entry Bias" and "Survivorship Bias"?

This is a question I have always struggled with - suppose you have medical data on patients over a period of time. This includes information on how long they spent in different states: Admission, ...
• 5,892
40 views

### Can some Survival Models "Dominate" other Survival Models?

I recently heard an interesting interpretation of Survival Models : A "standard" Survival Analysis problem (e.g. where at the end of the study, observations can either be "Censored"...
• 5,892
29 views

### Showing that random process is stationary

Suppose i have $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which positive ...
• 101
53 views

### Why do we need to Define "Valid" State Transitions in a Multi-State Model?

I was watching this video (https://www.youtube.com/watch?v=Wy-WmY6x4tg) and the presenter mentions (@ 8:10) that in a Multi-State Model, the user is required to specify number of "States" ...
• 5,892
108 views

### Markov Chain Monte Carlo with known normalisation

I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
• 141
1 vote
23 views

### Expectation of transition in a Markov process

I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C: \begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\ 0.0265225 & ...
• 113
1 vote
15 views

### How to map a sequence to a transition matrix

I have the following transition matrices, one for Maria and one for Anna: ...
• 391
11 views

• 151
1 vote
8 views

### Can we do Deep Reinforcement Learning with Disjoint Action Sets?

I'm defining a construction you can apply to a Markov decision process*, and it involves extending an equivalence relation from the the state space of the MDP to an equivalence relation on the action ...
• 11
11 views

### How to generate states for a Markov-Model through K-Means-Clustering on time-series?

I am reading a paper by Zufferey et al.: https://ieeexplore.ieee.org/document/8442470 On page 2 it says: "In this paper, the definition of the states is based on a K-Means clustering which allows ...
10 views

### Testing for Markov Process

Assume you have a box that has a button and a display panel. Every time the button is pressed an integer between 1 and N is displayed. The problem is to test which of two hypothesis is more likely: ...
64 views

### Can we rank markov matrices in stochastic order?

I am familiar with the concept of stochastic ordering for two random variables and how we can say if a markov matrix is stochastically monotone. What im interested in is if there is a concept for ...
• 792
126 views

### Markov chain and conditional probabilities

If I define a Markov chain, $$A\to B\to C \\P(A,B,C)=P(A)P(B|A)P(C|B)$$ Can I derive an expression for $P(C|A)$? I feel like this should be $P(C|B)P(B|A)$, but I am not seeing how to prove it.
• 133
1 vote
18 views

### Markov Chains: Do different transition probability matrices imply different stationary distributions?

As in the title. We know that irreducible Markov chains have a invariant distribution which is reached after enough steps (Reference: Billingsley's Probability and Measure, Theorem 8.6). Question: If ...
• 11
95 views

### Closed form for a markov chain, where transition probabilities depend on $n$?

Let $A_n$ be an event whose success probability $P(A_n)$ depends on the size, $n\in \mathbb{N}$, of a given sample. Let $d$ be a fixed positive integer. For the problem I am working on I can assume a ...
• 742
7 views

### Some hint for modeling a dynamic with Markov chain?

Consider a kind of game in which there is one card supplier and $n$ player $P=\{p_1,p_2,...p_n\}$ which consumes the cards. suppose the rules of the game are as follows: Card supplier supplies cards ...
1 vote
15 views

• 101
20 views

### How do HHMM (Hierarchical Hidden Markov Models) work and where to learn more?

I recently came across something called Hierarchical Hidden Markov Models. I am familiar with HMMs, but not HHMMs. I have two questions. I can't fully understand the procedure after reading here: ...
• 1,159