Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,189
questions
3
votes
2
answers
53
views
How could I estimate a transition probability matrix that varies over time?
I have multiple Markov chains with twelve states. I want to estimate a transition probability matrix for each time point (except for the last time point) that can vary over time using all Markov ...
- 55
0
votes
0
answers
8
views
Attribution modelling using First and Higher-Order Markov Chains
The crux of my question is as follows: Would a higher-order Markov model produce a different result than a first-order Markov model when used for Channel Attribution modelling?
Once the transition ...
- 115
0
votes
0
answers
5
views
Main event time prediction based on different sub events
As the title says, I want to predict the time (with a wide error range) of a main event’s first occurrence based on previous sub events that are vary in importance. These previous ‘predictor’ events ...
0
votes
0
answers
25
views
stationary distribution of a continuous time markov chain
With a certain rate $R$ balls fall into a box. There is no limit to the number of balls the box can hold, but each ball has a rate $\gamma$ to leave the box and when two balls hit each other they ...
- 1
0
votes
0
answers
23
views
Numerically stable computation of the variance [duplicate]
Suppose I've sampled $x_0,\ldots,x_{n-1}$ and want to calculate the variance of these samples. What is a good (numerically stable) algorithm for this? And does the answer change, if we impose ...
- 419
3
votes
1
answer
45
views
What kind of transition probability matrix indicates dependence/independence?
Suppose we have two discrete random variables $X$ and $Y$, both of which take values from $\{1,2,...,k\}$.
$Y$ is generated from $X$ via a transition probability matrix (also known as the stochastic ...
- 189
0
votes
0
answers
25
views
How are differential equations related to Markov processes?
How can I classify the two terms differential equation and Markov process? Here are a few questions that I ask myself:
Is a Markov process a superset of differential equation or vice versa?
Do ...
- 157
0
votes
0
answers
10
views
invariant distribution in discrete time markov chain and continuous time markov chain
The jump chain of a continuous time markov chain is a discrete time markov chain. I know that the existence of an invariant distribution for one chain does not imply the existence of an invariant ...
- 43
0
votes
1
answer
29
views
Determine recurrent states via First Return Probability
I have the following question. It's about transient and recurrent states in Markov Chains. I know when a state is one or the other, but there is one thing I can't figure out or understand.
We have ...
- 11
1
vote
0
answers
25
views
In Markov process, how to solve $P(X_n=1)$
Let $X_n$ is Markov chain for $n\geq 0$ and state space is $E=\{1,2,3\}$. One-step transition probability matrix is
$$
\left[ \begin{matrix}
p_{11}& p_{12}& p_{13}\\
p_{21}& p_{22}&...
- 11
0
votes
0
answers
19
views
Web Service Markov Chain - state probability calculation
I have the following problem:
Let us consider a Web server software that fails at the failure rate gp, running on a machine (node) that fails independently at the failure rate gm. An automatic failure ...
1
vote
1
answer
40
views
Estimating Markov chain transition probabilities from data
In a discrete time and space Markov chain, I know the formula to estimate the transition probabilities $$p_{ij} = \frac{n_{ij}}{\sum_{j \in S} n_{ij}}$$ I'm not sure however how you can find this is ...
- 13
0
votes
0
answers
2
views
Is there a way to learn/mine a process from continuous values and no actions?
I have following data:
value1, value2, valuen, reward
0.2, -0.2, 3.0, 0.22
..., ..., ..., ...
I would like to mine a process from this where I can find most ...
- 259
1
vote
1
answer
66
views
How can I simulate the stationary distribution of particles that each moves differently?
Suppose a particle enters a system at $0.5$ in the unit interval $[0,1]$.
With some probability $\lambda_{right}$, particles go right by
$$x_{right} = \frac{x\pi_{H}}{x\pi_{H} + (1-x)\pi_{L} }$$
and ...
- 193
3
votes
1
answer
112
views
Transition probabilities in multi dimensional birth-death process
I've got an urn problem that I believe can be nicely modeled as a birth-death process (I'm very new to markov chains, so maybe this is simply the wrong approach). Suppose we have $k$ urns each of ...
- 133
1
vote
1
answer
57
views
Does an expectation for a Markov chain simplify like this?
Let $x_0 \sim N(\mu_0,\Sigma_0)$, where $\mu_0 \in \mathbb R^n$ and $\Sigma_0 \in \mathbb R^{n \times n}$. Then, for $k = 0,\dots,N-1$, let $$x_{k+1} = Ax_k$$ for $A \in \mathbb R^{n \times n}$. Next, ...
- 3,657
6
votes
2
answers
98
views
How Do I Know If A Markov Chain Follows The Markov Property?
In Probability and Statistics, a Markov Chain is said to have the "Markov Property" if the next state of this Markov Chain only depends on the current state of this Markov Chain.
This being ...
- 7,028
1
vote
0
answers
23
views
How can i identify wether a Markov Chain is irreducible?
i am working on understanding Markov-Chains but i have a hard time understanding an irreducible MC. At our uni we have following definitions:
"If the set of all states is closed and does not ...
- 11
1
vote
0
answers
16
views
How to smoothen state probabilistic time series?
I have an array of where each columns represents the probability of being in a certain state and rows represent time indexes. Each one of the rows sum up to one (we always are in one state).
\begin{...
- 11
1
vote
0
answers
26
views
How to calculate a path entropy in the context of information theory?
Consider a path where the system goes from states $i$ to $j$ and then from $j$ to $k$. Given that the probability of finding the system in state $i$ is $p_i$ and the transition probabilities are $p_{...
- 61
0
votes
1
answer
37
views
Existence of the optimal control in finite horizon MDP
For infinite horizon MDP with finite state and action space, there exists an optimal (stationary) policy. For finite horizon MDP with finite state and action space, does there exist an optimal policy? ...
- 21
1
vote
1
answer
43
views
Queue M/M/2, proportion of unserved customers
In my work, I managed to arrive at the following scenario: Consider that I have a queue with two servers whose arrival rate is a poisson process $\lambda = 2 $ customers per hour and the service time ...
- 110
1
vote
0
answers
35
views
Probability of a "successful" arrival over time from $0$ to $t$?
I have this problem that I have been trying to solve using a random process (such is the task).
There are $n$ keys. Only one of them opens the lock. The keys are tried without replacement. Find the ...
- 11
1
vote
0
answers
30
views
How can I derive this equation?
I am reading Understanding Diffusion Models: A Unified Perspective, but I am confused about equation 45. Concretely, the consistency term:
$$
\sum_{t=1}^{T-1} \mathbb{E}_{q(x_{t-1},x_t,x_{t+1}|x_0)}\...
- 121
0
votes
0
answers
108
views
How To Estimate Markov Chain Probabilities in Real Life? [duplicate]
A question that I have always wondered about is that how are the Transition Probabilities within a Markov Chain estimated in real-world applications?
I tried to learn more about this online and found ...
- 7,028
1
vote
1
answer
40
views
Does large mutual information (between observations and parameter) imply the existence of a good estimator?
This question concerns the standard setting for applying Fano's inequality to derive minimax bounds for a parameter estimation problem.
The goal is to estimate a parameter described by a random ...
- 163
2
votes
0
answers
62
views
Training an autoregressive model using multiple independent sequences
I have a number of independent observed sequences that I believe are generated by same underlying process.
Is it possible to extend linear autoregression and learn the optimal transition matrix for ...
- 81
0
votes
0
answers
22
views
Bayes update rule studied as an operator
The bayes rule can be understood as a nonlinear map from the space of probability measures to itself. Are there any reference/books which study it from this perspective? For eg. can we say something ...
- 1
1
vote
0
answers
47
views
How to calculate the coefficient of absolute regularity?
I am reading the paper and try to understand the example solution in p. 14.
In particular, if the Markov chain has stationary distribution $\pi$ and $a$-step transition distribution $P^a$, then
$$β(a)...
- 850
0
votes
0
answers
20
views
How to estimate the order of a controlled Markov process from data?
Consider a non-stationary controlled Markov process represented by a sequence of states and actions $(s_0,a_0,\ldots,s_{T-1},a_{T-1},s_T)$ over a finite number of discrete time steps.
If the process ...
- 151
0
votes
0
answers
23
views
Aperiodic Markov Chains: If one state is aperiodic, does that mean all other states will also be aperiodic?
I was reading on Periodicity of Markov Chains, and while I understand that in order for a chain to be aperiodic it must be irreducible and all its states must be aperiodic, I was wondering if in the ...
- 1
0
votes
0
answers
18
views
How to apply exponentially decaying weights to converge PD estimates after 4-5 (or 7-10) years?
I have conditional Probability of Default (PD) estimates for 5 risk grades and for 6 year horizon using Markov Chain. I need to calibrate the first year to long run average and therefore change the ...
0
votes
0
answers
45
views
Continuous Time Markov Chain Generator Matrix
I'm looking for references on Continuous Time Markov Chain, with clearer explanation on generator matrix. Also, if there's large amount of examples that would be very helpful. Ross's book is very good....
- 6,174
0
votes
0
answers
12
views
Distribution of number of offsprings at n=10 (Galton Watson Process)
I have the following branching process problem.
Let $Z_n$ denote the population size at time $n$. It takes on the following probabilities:
...
0
votes
0
answers
23
views
Markov chain probability distribution
Given a Markov chain $X_1 \rightarrow X_2 \rightarrow \dotsc \rightarrow X_N$ we can write
$$
p(x_1, x_2, \dotsc, x_N) = p(x_1 \mid x_2) \cdot p(x_2 \mid x_3) \cdot \dotsc \cdot p(x_{N-1}, x_N)
$$
How ...
- 637
1
vote
0
answers
114
views
help me understand a part of the baum welch algorithm for hidden markov models
I am having troubles understanding a crucial part of the baum-welch algorithm in hidden markov models.
When we calculate zhe/digamma representing the probability of being in state i at timestep t and ...
0
votes
1
answer
58
views
Could the likelihood increase monotonically in a misspecified EM algorithm?
I am dealing with the estimation of a Gaussian Hidden Markov Model with conditional distribution given the first-order Markov state $S_t = j,\ j=1,...,J$
$$
Y_t|S_t=j\sim N(0,\sigma^2_j)
$$
where the ...
- 319
0
votes
1
answer
96
views
Creating synthetic data for time series, Hidden Markov Model
Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
- 288
1
vote
1
answer
41
views
Markov chain: inferring transition rates from equilibrium
I feel like I have been having a very dumb week trying to solve/research this problem and that I am missing an easy solution or that it is not possible. Given an equilibrium distribution (say from a ...
- 11
3
votes
1
answer
189
views
Why do we sample from the uniform distribution in Metropolis-Hastings for acceptance?
For each iteration of the MH, sample $x'=q(x|x')$, then the acceptance probability is computed:$$A=\min(1,a)$$
where
$$
\alpha=\frac{p(x')q(x|x')}{p(x)q(x'|x)}
$$
Now, I've seen that the algorithm ...
1
vote
0
answers
79
views
How to prove a sequence of random variables dependent on different random variables is a homogeneous Markov chain?
In Pierre Brémaud's book, Markov Chains - Gibbs Fields, Monte Carlo Simulation and Queues, exercise 2.6.9 is stated as follows:
Let $\{Z_n\}_{n \geq 1}$ be an IID sequence of geometric random ...
- 610
1
vote
0
answers
1k
views
"SVD did not converge" while using statsmodel.timeseriesanalysis.MarkovRegression
I am trying to fit a MarkovRegression model sm.tsa.MarkovRegression(spread, k_regimes=2, switching_variance=True) to a time series of price spread.
This is the ...
- 11
0
votes
0
answers
17
views
Is this Markov chain of a UAU-process (unaware - aware - unaware) convergent and why?
I am currently looking at a Markov chain of $UAU$-process on a uni-weighted undirected network. Where individuals are aware of certain arbitrary information or not. The individuals are the nodes of ...
0
votes
0
answers
67
views
For markov transition matrix and the initial state, calculate probability to reach certain other state in k or less steps
So there is a vector n giving the initial state and a Markov transition probability matrix M. I know I can calculate the ...
- 71
0
votes
1
answer
31
views
Stationary Distributions and Financial Markets
Not sure my idea makes sense but here goes. Financial markets are notoriously hard to model. Is this perhaps there does not exist a stationary distribution for any Markov chain stochastic process ...
4
votes
2
answers
304
views
Understanding the Difference Between Different Types of Markov Chains?
I have been trying to learn more about different types of Markov Chains. So far, here is my basic understanding of them:
Discrete Time Markov Chain: Characterized by a constant transition probability ...
- 7,028
1
vote
1
answer
46
views
What is the influence of initial state in sequence generated from a markov chain?
For thousands of item, I have observations about their state (a letter) for 9 timestep.
From that, I build a transition matrix (RotationMatrix by couting their ...
- 51
0
votes
1
answer
284
views
About the paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"
I have spent some time studying the paper Deep Unsupervised Learning using
Nonequilibrium Thermodynamics. At page 5, the authors discuss the following integral:
$$\int d\mathbf{x}^{(1\cdots T)}q(\...
- 21
1
vote
1
answer
52
views
How to use Stirling's formula of n! in this probability computations in random walk?
I want to compute $ \binom{2n}{n} p^n (1-p)^n = \frac{(2n)!}{n!n!}(p(1-p))^n, n=1,2,3...$
By using an approximation, due to Stirling, which asserts $ n! \sim n^{(n +\frac12)}e^{-n}\sqrt{2\pi}$
Where ...
- 1,140
0
votes
0
answers
33
views
Chapman-Kolmogorov equations (Markov Chain)
For a Markov chain $\{X_n, n \geqslant 0 \}$ with transition probabilities $P_{i,j},$ consider the conditional probability that $X_n =m$ given that the chain started at time 0 in state i and has not ...
- 1,140