Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,164
questions
-1
votes
0
answers
10
views
Logistic regression(with Markov chain) on time series data?
I'm working with a biotech device's time series data to predict the replacement amount.
The background is the battery of the device will die after the implant for a few years, and the battery will be ...
- 1
0
votes
1
answer
19
views
Creating synthetic data for time series, Hidden Markov Model
Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
- 269
1
vote
1
answer
18
views
Markov chain: inferring transition rates from equilibrium
I feel like I have been having a very dumb week trying to solve/research this problem and that I am missing an easy solution or that it is not possible. Given an equilibrium distribution (say from a ...
- 11
2
votes
1
answer
90
views
Why do we sample from the uniform distribution in Metropolis-Hastings for acceptance?
For each iteration of the MH, sample $x'=q(x|x')$, then the acceptance probability is computed:$$A=\min(1,a)$$
where
$$
\alpha=\frac{p(x')q(x|x')}{p(x)q(x'|x)}
$$
Now, I've seen that the algorithm ...
- 487
1
vote
0
answers
18
views
How to prove a sequence of random variables dependent on different random variables is a homogeneous Markov chain?
In Pierre Brémaud's book, Markov Chains - Gibbs Fields, Monte Carlo Simulation and Queues, exercise 2.6.9 is stated as follows:
Let $\{Z_n\}_{n \geq 1}$ be an IID sequence of geometric random ...
- 648
1
vote
0
answers
69
views
"SVD did not converge" while using statsmodel.timeseriesanalysis.MarkovRegression
I am trying to fit a MarkovRegression model sm.tsa.MarkovRegression(spread, k_regimes=2, switching_variance=True) to a time series of price spread.
This is the ...
- 11
0
votes
0
answers
17
views
Is this Markov chain of a UAU-process (unaware - aware - unaware) convergent and why?
I am currently looking at a Markov chain of $UAU$-process on a uni-weighted undirected network. Where individuals are aware of certain arbitrary information or not. The individuals are the nodes of ...
0
votes
0
answers
14
views
For markov transition matrix and the initial state, calculate probability to reach certain other state in k or less steps
So there is a vector n giving the initial state and a Markov transition probability matrix M. I know I can calculate the ...
- 71
0
votes
1
answer
21
views
Stationary Distributions and Financial Markets
Not sure my idea makes sense but here goes. Financial markets are notoriously hard to model. Is this perhaps there does not exist a stationary distribution for any Markov chain stochastic process ...
3
votes
0
answers
63
views
Understanding the Difference Between Different Types of Markov Chains?
I have been trying to learn more about different types of Markov Chains. So far, here is my basic understanding of them:
Discrete Time Markov Chain: Characterized by a constant transition probability ...
- 6,048
1
vote
1
answer
32
views
What is the influence of initial state in sequence generated from a markov chain?
For thousands of item, I have observations about their state (a letter) for 9 timestep.
From that, I build a transition matrix (RotationMatrix by couting their ...
- 51
0
votes
1
answer
67
views
About the paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"
I have spent some time studying the paper Deep Unsupervised Learning using
Nonequilibrium Thermodynamics. At page 5, the authors discuss the following integral:
$$\int d\mathbf{x}^{(1\cdots T)}q(\...
- 21
1
vote
1
answer
38
views
How to use Stirling's formula of n! in this probability computations in random walk?
I want to compute $ \binom{2n}{n} p^n (1-p)^n = \frac{(2n)!}{n!n!}(p(1-p))^n, n=1,2,3...$
By using an approximation, due to Stirling, which asserts $ n! \sim n^{(n +\frac12)}e^{-n}\sqrt{2\pi}$
Where ...
- 1,118
0
votes
0
answers
21
views
Chapman-Kolmogorov equations (Markov Chain)
For a Markov chain $\{X_n, n \geqslant 0 \}$ with transition probabilities $P_{i,j},$ consider the conditional probability that $X_n =m$ given that the chain started at time 0 in state i and has not ...
- 1,118
0
votes
0
answers
22
views
Obtaining log-likelihood value of a Markov chain when probability transition matrix contains exact-zero entries
I have a $n$ sequences $\boldsymbol{X}_1, \dots, \boldsymbol{X}_n$ of varying lengths arising from a Markov chain with a large state space $\mathcal{S} = \{ 1, 2, \dots, s \}$. Suppose the initial ...
- 11
0
votes
0
answers
26
views
Anomaly detection on a symbolic sequence using markov chain
I want to do anomaly detection on time series data that I have converted to a symbolic sequence. I want to try this using a markov chain.
Lets say I have symbols $A$ and $B$. A markov chain $M$ will ...
- 21
0
votes
0
answers
17
views
Markov's Decision Process - calculate value in each iteration
I have the following decision tree:
I calculated the value of the plan using the following paramenters (given):
{𝑆0 → 𝑎1 , 𝑆1 → 𝑎3 , 𝑆2 → 𝑎4 },
Discount factor (𝛾)= 0.2
I used this formula to ...
- 11
1
vote
1
answer
85
views
Coin flipping (Markov chain)
Suppose that coin 1 has probability 0.7 of coming up heads, and coin 2 has probability 0.6 of coming up heads.
If the coin flipped today comes up heads, then we select coin 1 to flip tomorrow, and if ...
- 1,118
0
votes
0
answers
43
views
Is this Markov chain? If not, how can we transform it to Markov chain?
Consider a process $\{ X_n, n= 0,1,...\},$ which takes on values 0,1 or 2. Suppose $$P\{X_{n+1}=j| X_n=i, X_{n-1}=i_{n-1},...,X_0= i_0 \} = \begin{cases} P^{\rm{I}}_{ij}, \text{when n} \ \text{is even,...
- 1,118
0
votes
0
answers
14
views
Correlation of entries of the fundamental matrix of an absorbing Markov chain
Suppose I have an absorbing Markov chain with state space $S$, partitioned into $T$, the set of transient states, and $A$, the set of absorbing states. Let $N$ denote the fundamental matrix of this ...
- 1,002
0
votes
3
answers
72
views
Why not just run a Markov chain to get stationary probabilities?
I'm reading Performance Modeling and Design of Computer Systems which contains some analysis of Markov chains. In particular, it emphasises various analytical methods for finding the stationary ...
- 759
0
votes
0
answers
19
views
How to decide $X_n, n \in [0,\infty)$ is a Markov chain and how to compute transition probability matrix?
Three white and three black balls are distributed in two urns in such a way that each contains three balls. We say that the system is in state i, i = 0, 1, 2, 3, if the first urn contains i white ...
- 1,118
0
votes
0
answers
29
views
Is this transition probability matrix correct?
A pensioner receives 2 (thousand dollars) at the beginning of each month. The amount of money he needs to spend during a month is independent of the amount he has and is equal to i with probability $...
- 1,118
0
votes
0
answers
30
views
How to use here Markov chain, to compute the required probability?
Suppose that balls are successively distributed among 8 urns, with each ball being equally likely to be put in any of these urns. What is the probability that there will be exactly 3 nonempty urns ...
- 1,118
0
votes
0
answers
12
views
sum of expected differences in time series
I have a (markov) decision process without reward.
I am estimating the expected state differences $\mathbb{E}[\delta_t]$ where $\delta_t = X_{t+1} - X_t$.
A state $X_{t+1}$ can be expressed as $X_0 + \...
- 101
1
vote
1
answer
50
views
Generate conditional probability matrix for simulation
I want to generate a categorical random variable conditioned on other variables, for example, generate $X$ from $Y$ by $P(X|Y)$, or generate $X$ from $Y_1,Y_2$ by $P(X|Y_1,Y_2)$.
Here, $P(X|Y)$ ...
- 137
0
votes
0
answers
16
views
Reinforcement Learning MDP stochastic policies
I am struggling to get an understanding behind stochastic policy in a MDP
I was reading this paper and in section 3.2 they say
At each time step $t$, corresponding to extracting
sentence number $t$, ...
0
votes
0
answers
13
views
How to define relative value functions for multichain MDPs
I have been studying the average reward MDP, and I found that in most references, the average reward criterion is defined as $\rho(s)=\lim_{N\to\infty} E_\pi[\frac{1}{N}\sum^{N-1}_{t=0}r_\pi(s_t)]$, ...
- 101
0
votes
0
answers
23
views
How to test for equality two finite state markov chains?
I have the transition probabilities of the two markov chains. I think that maybe I could apply the Kullback–Leibler divergence two the stationary distributions of these Markov chains (if they are ...
- 1,400
0
votes
0
answers
28
views
Data Sufficiency for Markov Model
I have a question regarding the Markov model. Suppose I have N number of data in a table and I constructed a simple Markov model with their state transitions as shown below.
How do I know if I have &...
6
votes
2
answers
569
views
Can any Models be "Bagged"?
I have been learning about "bagging" (bootstrap aggregation) - supposedly, there are many types of statistical models can be bagged together. For example, CART Decision Trees can be "...
- 6,048
1
vote
0
answers
24
views
Memorylessness by way of additional dimensions
This is a somewhat broad question that occurred to me regarding the nature of memorylessness. Namely: Is there utility in considering systems which are themselves not memoryless, but then expanding ...
- 261
1
vote
1
answer
26
views
Goodness of fit test for a transition density function of a Markov process
Suppose that you have one realization $x = \{x_n\}_{n = 1}^{N}$ of the stochastic process $X = \{X_n\}_{n = 1}^{N}$ with state space $\mathbb{R}$. Assume that the process is Markovian, time-...
- 11
1
vote
0
answers
12
views
Coding assignment autograder: Checking if a data stream is generated by a particular Markov process
I teach an introductory programming course in which most assignments ship with a set of tests students can use locally on their machine to validate that their code works as intended.
Many years ago we ...
- 136
0
votes
0
answers
16
views
continuous time markov chain sample size
Is that the same if I have one patient with one Markov chain of length 100, and 50 patients with 50 Markov chains of length 2?
If they are the same, why?
Thanks
- 51
0
votes
1
answer
45
views
Balancing "Delayed Entry Bias" and "Survivorship Bias"?
This is a question I have always struggled with - suppose you have medical data on patients over a period of time. This includes information on how long they spent in different states: Admission, ...
- 6,048
0
votes
1
answer
49
views
Can some Survival Models "Dominate" other Survival Models?
I recently heard an interesting interpretation of Survival Models : A "standard" Survival Analysis problem (e.g. where at the end of the study, observations can either be "Censored"...
- 6,048
0
votes
0
answers
31
views
Showing that random process is stationary
Suppose i have $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which positive ...
- 101
2
votes
1
answer
59
views
Why do we need to Define "Valid" State Transitions in a Multi-State Model?
I was watching this video (https://www.youtube.com/watch?v=Wy-WmY6x4tg) and the presenter mentions (@ 8:10) that in a Multi-State Model, the user is required to specify number of "States" ...
- 6,048
4
votes
1
answer
119
views
Markov Chain Monte Carlo with known normalisation
I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
- 141
1
vote
1
answer
41
views
Expectation of transition in a Markov process
I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C:
\begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\
0.0265225 & ...
- 123
1
vote
0
answers
28
views
How to map a sequence to a transition matrix
I have the following transition matrices, one for Maria and one for Anna:
...
- 391
0
votes
0
answers
16
views
The gamble problem to find limiting stationary distribution
[This question was asked here before, but I did not get satisfactory response]
I was trying to solve the following problem:
I find out the transition matrix is:
$$P = \begin{bmatrix}
1 & 0 &0 ...
- 243
0
votes
0
answers
12
views
How does detailed balance relate to (conditional) expectation?
Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
- 1,560
0
votes
0
answers
22
views
First order PDF of a stochastic process
I've started studying about stochastic processes and I need some help in this question.
A random number generator is making numbers by this process:
First number (X0) is a sample from Normal Standard ...
- 1
1
vote
1
answer
75
views
What type of Markov Chain is a random walk of a Knight on a chessboard?
Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
- 129
2
votes
0
answers
29
views
Hidden Markov Model observing sequences
I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
- 143
0
votes
0
answers
75
views
Prove stationary distribution of the CTMC with Cut method
Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$
on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition)
($A$, $A^c$
) with $A ⊂ S$:...
- 1
0
votes
0
answers
27
views
Identification of the transition probability of a time homogeneous MDP with subsampling
I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation.
...
- 137
0
votes
0
answers
30
views
Why the memoryless Markov Property is desirable
The memoryless Markov property says future predictions only depend on the current status.
With longitudinal data, we have all the past data recorded. Why cannot we make use of all the info? Why should ...
- 51