Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
3 votes
2 answers
53 views

How could I estimate a transition probability matrix that varies over time?

I have multiple Markov chains with twelve states. I want to estimate a transition probability matrix for each time point (except for the last time point) that can vary over time using all Markov ...
0 votes
0 answers
8 views

Attribution modelling using First and Higher-Order Markov Chains

The crux of my question is as follows: Would a higher-order Markov model produce a different result than a first-order Markov model when used for Channel Attribution modelling? Once the transition ...
0 votes
0 answers
5 views

Main event time prediction based on different sub events

As the title says, I want to predict the time (with a wide error range) of a main event’s first occurrence based on previous sub events that are vary in importance. These previous ‘predictor’ events ...
0 votes
0 answers
25 views

stationary distribution of a continuous time markov chain

With a certain rate $R$ balls fall into a box. There is no limit to the number of balls the box can hold, but each ball has a rate $\gamma$ to leave the box and when two balls hit each other they ...
  • 1
0 votes
0 answers
23 views

Numerically stable computation of the variance [duplicate]

Suppose I've sampled $x_0,\ldots,x_{n-1}$ and want to calculate the variance of these samples. What is a good (numerically stable) algorithm for this? And does the answer change, if we impose ...
  • 419
3 votes
1 answer
45 views

What kind of transition probability matrix indicates dependence/independence?

Suppose we have two discrete random variables $X$ and $Y$, both of which take values from $\{1,2,...,k\}$. $Y$ is generated from $X$ via a transition probability matrix (also known as the stochastic ...
0 votes
0 answers
25 views

How are differential equations related to Markov processes?

How can I classify the two terms differential equation and Markov process? Here are a few questions that I ask myself: Is a Markov process a superset of differential equation or vice versa? Do ...
  • 157
0 votes
0 answers
10 views

invariant distribution in discrete time markov chain and continuous time markov chain

The jump chain of a continuous time markov chain is a discrete time markov chain. I know that the existence of an invariant distribution for one chain does not imply the existence of an invariant ...
0 votes
1 answer
29 views

Determine recurrent states via First Return Probability

I have the following question. It's about transient and recurrent states in Markov Chains. I know when a state is one or the other, but there is one thing I can't figure out or understand. We have ...
1 vote
0 answers
25 views

In Markov process, how to solve $P(X_n=1)$

Let $X_n$ is Markov chain for $n\geq 0$ and state space is $E=\{1,2,3\}$. One-step transition probability matrix is $$ \left[ \begin{matrix} p_{11}& p_{12}& p_{13}\\ p_{21}& p_{22}&...
  • 11
0 votes
0 answers
19 views

Web Service Markov Chain - state probability calculation

I have the following problem: Let us consider a Web server software that fails at the failure rate gp, running on a machine (node) that fails independently at the failure rate gm. An automatic failure ...
1 vote
1 answer
40 views

Estimating Markov chain transition probabilities from data

In a discrete time and space Markov chain, I know the formula to estimate the transition probabilities $$p_{ij} = \frac{n_{ij}}{\sum_{j \in S} n_{ij}}$$ I'm not sure however how you can find this is ...
0 votes
0 answers
2 views

Is there a way to learn/mine a process from continuous values and no actions?

I have following data: value1, value2, valuen, reward 0.2, -0.2, 3.0, 0.22 ..., ..., ..., ... I would like to mine a process from this where I can find most ...
  • 259
1 vote
1 answer
66 views

How can I simulate the stationary distribution of particles that each moves differently?

Suppose a particle enters a system at $0.5$ in the unit interval $[0,1]$. With some probability $\lambda_{right}$, particles go right by $$x_{right} = \frac{x\pi_{H}}{x\pi_{H} + (1-x)\pi_{L} }$$ and ...
3 votes
1 answer
112 views

Transition probabilities in multi dimensional birth-death process

I've got an urn problem that I believe can be nicely modeled as a birth-death process (I'm very new to markov chains, so maybe this is simply the wrong approach). Suppose we have $k$ urns each of ...
  • 133
1 vote
1 answer
57 views

Does an expectation for a Markov chain simplify like this?

Let $x_0 \sim N(\mu_0,\Sigma_0)$, where $\mu_0 \in \mathbb R^n$ and $\Sigma_0 \in \mathbb R^{n \times n}$. Then, for $k = 0,\dots,N-1$, let $$x_{k+1} = Ax_k$$ for $A \in \mathbb R^{n \times n}$. Next, ...
  • 3,657
6 votes
2 answers
98 views

How Do I Know If A Markov Chain Follows The Markov Property?

In Probability and Statistics, a Markov Chain is said to have the "Markov Property" if the next state of this Markov Chain only depends on the current state of this Markov Chain. This being ...
  • 7,028
1 vote
0 answers
23 views

How can i identify wether a Markov Chain is irreducible?

i am working on understanding Markov-Chains but i have a hard time understanding an irreducible MC. At our uni we have following definitions: "If the set of all states is closed and does not ...
1 vote
0 answers
16 views

How to smoothen state probabilistic time series?

I have an array of where each columns represents the probability of being in a certain state and rows represent time indexes. Each one of the rows sum up to one (we always are in one state). \begin{...
  • 11
1 vote
0 answers
26 views

How to calculate a path entropy in the context of information theory?

Consider a path where the system goes from states $i$ to $j$ and then from $j$ to $k$. Given that the probability of finding the system in state $i$ is $p_i$ and the transition probabilities are $p_{...
  • 61
0 votes
1 answer
37 views

Existence of the optimal control in finite horizon MDP

For infinite horizon MDP with finite state and action space, there exists an optimal (stationary) policy. For finite horizon MDP with finite state and action space, does there exist an optimal policy? ...
1 vote
1 answer
43 views

Queue M/M/2, proportion of unserved customers

In my work, I managed to arrive at the following scenario: Consider that I have a queue with two servers whose arrival rate is a poisson process $\lambda = 2 $ customers per hour and the service time ...
  • 110
1 vote
0 answers
35 views

Probability of a "successful" arrival over time from $0$ to $t$?

I have this problem that I have been trying to solve using a random process (such is the task). There are $n$ keys. Only one of them opens the lock. The keys are tried without replacement. Find the ...
1 vote
0 answers
30 views

How can I derive this equation?

I am reading Understanding Diffusion Models: A Unified Perspective, but I am confused about equation 45. Concretely, the consistency term: $$ \sum_{t=1}^{T-1} \mathbb{E}_{q(x_{t-1},x_t,x_{t+1}|x_0)}\...
  • 121
0 votes
0 answers
108 views

How To Estimate Markov Chain Probabilities in Real Life? [duplicate]

A question that I have always wondered about is that how are the Transition Probabilities within a Markov Chain estimated in real-world applications? I tried to learn more about this online and found ...
  • 7,028
1 vote
1 answer
40 views

Does large mutual information (between observations and parameter) imply the existence of a good estimator?

This question concerns the standard setting for applying Fano's inequality to derive minimax bounds for a parameter estimation problem. The goal is to estimate a parameter described by a random ...
  • 163
2 votes
0 answers
62 views

Training an autoregressive model using multiple independent sequences

I have a number of independent observed sequences that I believe are generated by same underlying process. Is it possible to extend linear autoregression and learn the optimal transition matrix for ...
  • 81
0 votes
0 answers
22 views

Bayes update rule studied as an operator

The bayes rule can be understood as a nonlinear map from the space of probability measures to itself. Are there any reference/books which study it from this perspective? For eg. can we say something ...
1 vote
0 answers
47 views

How to calculate the coefficient of absolute regularity?

I am reading the paper and try to understand the example solution in p. 14. In particular, if the Markov chain has stationary distribution $\pi$ and $a$-step transition distribution $P^a$, then $$β(a)...
  • 850
0 votes
0 answers
20 views

How to estimate the order of a controlled Markov process from data?

Consider a non-stationary controlled Markov process represented by a sequence of states and actions $(s_0,a_0,\ldots,s_{T-1},a_{T-1},s_T)$ over a finite number of discrete time steps. If the process ...
  • 151
0 votes
0 answers
23 views

Aperiodic Markov Chains: If one state is aperiodic, does that mean all other states will also be aperiodic?

I was reading on Periodicity of Markov Chains, and while I understand that in order for a chain to be aperiodic it must be irreducible and all its states must be aperiodic, I was wondering if in the ...
  • 1
0 votes
0 answers
18 views

How to apply exponentially decaying weights to converge PD estimates after 4-5 (or 7-10) years?

I have conditional Probability of Default (PD) estimates for 5 risk grades and for 6 year horizon using Markov Chain. I need to calibrate the first year to long run average and therefore change the ...
0 votes
0 answers
45 views

Continuous Time Markov Chain Generator Matrix

I'm looking for references on Continuous Time Markov Chain, with clearer explanation on generator matrix. Also, if there's large amount of examples that would be very helpful. Ross's book is very good....
  • 6,174
0 votes
0 answers
12 views

Distribution of number of offsprings at n=10 (Galton Watson Process)

I have the following branching process problem. Let $Z_n$ denote the population size at time $n$. It takes on the following probabilities: ...
0 votes
0 answers
23 views

Markov chain probability distribution

Given a Markov chain $X_1 \rightarrow X_2 \rightarrow \dotsc \rightarrow X_N$ we can write $$ p(x_1, x_2, \dotsc, x_N) = p(x_1 \mid x_2) \cdot p(x_2 \mid x_3) \cdot \dotsc \cdot p(x_{N-1}, x_N) $$ How ...
  • 637
1 vote
0 answers
114 views

help me understand a part of the baum welch algorithm for hidden markov models

I am having troubles understanding a crucial part of the baum-welch algorithm in hidden markov models. When we calculate zhe/digamma representing the probability of being in state i at timestep t and ...
0 votes
1 answer
58 views

Could the likelihood increase monotonically in a misspecified EM algorithm?

I am dealing with the estimation of a Gaussian Hidden Markov Model with conditional distribution given the first-order Markov state $S_t = j,\ j=1,...,J$ $$ Y_t|S_t=j\sim N(0,\sigma^2_j) $$ where the ...
  • 319
0 votes
1 answer
96 views

Creating synthetic data for time series, Hidden Markov Model

Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
  • 288
1 vote
1 answer
41 views

Markov chain: inferring transition rates from equilibrium

I feel like I have been having a very dumb week trying to solve/research this problem and that I am missing an easy solution or that it is not possible. Given an equilibrium distribution (say from a ...
3 votes
1 answer
189 views

Why do we sample from the uniform distribution in Metropolis-Hastings for acceptance?

For each iteration of the MH, sample $x'=q(x|x')$, then the acceptance probability is computed:$$A=\min(1,a)$$ where $$ \alpha=\frac{p(x')q(x|x')}{p(x)q(x'|x)} $$ Now, I've seen that the algorithm ...
1 vote
0 answers
79 views

How to prove a sequence of random variables dependent on different random variables is a homogeneous Markov chain?

In Pierre Brémaud's book, Markov Chains - Gibbs Fields, Monte Carlo Simulation and Queues, exercise 2.6.9 is stated as follows: Let $\{Z_n\}_{n \geq 1}$ be an IID sequence of geometric random ...
  • 610
1 vote
0 answers
1k views

"SVD did not converge" while using statsmodel.timeseriesanalysis.MarkovRegression

I am trying to fit a MarkovRegression model sm.tsa.MarkovRegression(spread, k_regimes=2, switching_variance=True) to a time series of price spread. This is the ...
0 votes
0 answers
17 views

Is this Markov chain of a UAU-process (unaware - aware - unaware) convergent and why?

I am currently looking at a Markov chain of $UAU$-process on a uni-weighted undirected network. Where individuals are aware of certain arbitrary information or not. The individuals are the nodes of ...
0 votes
0 answers
67 views

For markov transition matrix and the initial state, calculate probability to reach certain other state in k or less steps

So there is a vector n giving the initial state and a Markov transition probability matrix M. I know I can calculate the ...
  • 71
0 votes
1 answer
31 views

Stationary Distributions and Financial Markets

Not sure my idea makes sense but here goes. Financial markets are notoriously hard to model. Is this perhaps there does not exist a stationary distribution for any Markov chain stochastic process ...
4 votes
2 answers
304 views

Understanding the Difference Between Different Types of Markov Chains?

I have been trying to learn more about different types of Markov Chains. So far, here is my basic understanding of them: Discrete Time Markov Chain: Characterized by a constant transition probability ...
  • 7,028
1 vote
1 answer
46 views

What is the influence of initial state in sequence generated from a markov chain?

For thousands of item, I have observations about their state (a letter) for 9 timestep. From that, I build a transition matrix (RotationMatrix by couting their ...
0 votes
1 answer
284 views

About the paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"

I have spent some time studying the paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics. At page 5, the authors discuss the following integral: $$\int d\mathbf{x}^{(1\cdots T)}q(\...
  • 21
1 vote
1 answer
52 views

How to use Stirling's formula of n! in this probability computations in random walk?

I want to compute $ \binom{2n}{n} p^n (1-p)^n = \frac{(2n)!}{n!n!}(p(1-p))^n, n=1,2,3...$ By using an approximation, due to Stirling, which asserts $ n! \sim n^{(n +\frac12)}e^{-n}\sqrt{2\pi}$ Where ...
0 votes
0 answers
33 views

Chapman-Kolmogorov equations (Markov Chain)

For a Markov chain $\{X_n, n \geqslant 0 \}$ with transition probabilities $P_{i,j},$ consider the conditional probability that $X_n =m$ given that the chain started at time 0 in state i and has not ...

1
2 3 4 5
24