Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
10 views

Logistic regression(with Markov chain) on time series data?

I'm working with a biotech device's time series data to predict the replacement amount. The background is the battery of the device will die after the implant for a few years, and the battery will be ...
  • 1
0 votes
1 answer
19 views

Creating synthetic data for time series, Hidden Markov Model

Suppose that I have a task of classifying a time series. I decide to use Hidden Markov Model $\lambda(A, B, \pi)$, where $A$ is a transition matrix, $B$ is an emission probability, $\pi$ is an initial ...
  • 269
1 vote
1 answer
18 views

Markov chain: inferring transition rates from equilibrium

I feel like I have been having a very dumb week trying to solve/research this problem and that I am missing an easy solution or that it is not possible. Given an equilibrium distribution (say from a ...
2 votes
1 answer
90 views

Why do we sample from the uniform distribution in Metropolis-Hastings for acceptance?

For each iteration of the MH, sample $x'=q(x|x')$, then the acceptance probability is computed:$$A=\min(1,a)$$ where $$ \alpha=\frac{p(x')q(x|x')}{p(x)q(x'|x)} $$ Now, I've seen that the algorithm ...
  • 487
1 vote
0 answers
18 views

How to prove a sequence of random variables dependent on different random variables is a homogeneous Markov chain?

In Pierre Brémaud's book, Markov Chains - Gibbs Fields, Monte Carlo Simulation and Queues, exercise 2.6.9 is stated as follows: Let $\{Z_n\}_{n \geq 1}$ be an IID sequence of geometric random ...
  • 648
1 vote
0 answers
69 views

"SVD did not converge" while using statsmodel.timeseriesanalysis.MarkovRegression

I am trying to fit a MarkovRegression model sm.tsa.MarkovRegression(spread, k_regimes=2, switching_variance=True) to a time series of price spread. This is the ...
0 votes
0 answers
17 views

Is this Markov chain of a UAU-process (unaware - aware - unaware) convergent and why?

I am currently looking at a Markov chain of $UAU$-process on a uni-weighted undirected network. Where individuals are aware of certain arbitrary information or not. The individuals are the nodes of ...
0 votes
0 answers
14 views

For markov transition matrix and the initial state, calculate probability to reach certain other state in k or less steps

So there is a vector n giving the initial state and a Markov transition probability matrix M. I know I can calculate the ...
  • 71
0 votes
1 answer
21 views

Stationary Distributions and Financial Markets

Not sure my idea makes sense but here goes. Financial markets are notoriously hard to model. Is this perhaps there does not exist a stationary distribution for any Markov chain stochastic process ...
3 votes
0 answers
63 views

Understanding the Difference Between Different Types of Markov Chains?

I have been trying to learn more about different types of Markov Chains. So far, here is my basic understanding of them: Discrete Time Markov Chain: Characterized by a constant transition probability ...
  • 6,048
1 vote
1 answer
32 views

What is the influence of initial state in sequence generated from a markov chain?

For thousands of item, I have observations about their state (a letter) for 9 timestep. From that, I build a transition matrix (RotationMatrix by couting their ...
0 votes
1 answer
67 views

About the paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"

I have spent some time studying the paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics. At page 5, the authors discuss the following integral: $$\int d\mathbf{x}^{(1\cdots T)}q(\...
  • 21
1 vote
1 answer
38 views

How to use Stirling's formula of n! in this probability computations in random walk?

I want to compute $ \binom{2n}{n} p^n (1-p)^n = \frac{(2n)!}{n!n!}(p(1-p))^n, n=1,2,3...$ By using an approximation, due to Stirling, which asserts $ n! \sim n^{(n +\frac12)}e^{-n}\sqrt{2\pi}$ Where ...
0 votes
0 answers
21 views

Chapman-Kolmogorov equations (Markov Chain)

For a Markov chain $\{X_n, n \geqslant 0 \}$ with transition probabilities $P_{i,j},$ consider the conditional probability that $X_n =m$ given that the chain started at time 0 in state i and has not ...
0 votes
0 answers
22 views

Obtaining log-likelihood value of a Markov chain when probability transition matrix contains exact-zero entries

I have a $n$ sequences $\boldsymbol{X}_1, \dots, \boldsymbol{X}_n$ of varying lengths arising from a Markov chain with a large state space $\mathcal{S} = \{ 1, 2, \dots, s \}$. Suppose the initial ...
0 votes
0 answers
26 views

Anomaly detection on a symbolic sequence using markov chain

I want to do anomaly detection on time series data that I have converted to a symbolic sequence. I want to try this using a markov chain. Lets say I have symbols $A$ and $B$. A markov chain $M$ will ...
0 votes
0 answers
17 views

Markov's Decision Process - calculate value in each iteration

I have the following decision tree: I calculated the value of the plan using the following paramenters (given): {𝑆0 → 𝑎1 , 𝑆1 → 𝑎3 , 𝑆2 → 𝑎4 }, Discount factor (𝛾)= 0.2 I used this formula to ...
1 vote
1 answer
85 views

Coin flipping (Markov chain)

Suppose that coin 1 has probability 0.7 of coming up heads, and coin 2 has probability 0.6 of coming up heads. If the coin flipped today comes up heads, then we select coin 1 to flip tomorrow, and if ...
0 votes
0 answers
43 views

Is this Markov chain? If not, how can we transform it to Markov chain?

Consider a process $\{ X_n, n= 0,1,...\},$ which takes on values 0,1 or 2. Suppose $$P\{X_{n+1}=j| X_n=i, X_{n-1}=i_{n-1},...,X_0= i_0 \} = \begin{cases} P^{\rm{I}}_{ij}, \text{when n} \ \text{is even,...
0 votes
0 answers
14 views

Correlation of entries of the fundamental matrix of an absorbing Markov chain

Suppose I have an absorbing Markov chain with state space $S$, partitioned into $T$, the set of transient states, and $A$, the set of absorbing states. Let $N$ denote the fundamental matrix of this ...
  • 1,002
0 votes
3 answers
72 views

Why not just run a Markov chain to get stationary probabilities?

I'm reading Performance Modeling and Design of Computer Systems which contains some analysis of Markov chains. In particular, it emphasises various analytical methods for finding the stationary ...
  • 759
0 votes
0 answers
19 views

How to decide $X_n, n \in [0,\infty)$ is a Markov chain and how to compute transition probability matrix?

Three white and three black balls are distributed in two urns in such a way that each contains three balls. We say that the system is in state i, i = 0, 1, 2, 3, if the first urn contains i white ...
0 votes
0 answers
29 views

Is this transition probability matrix correct?

A pensioner receives 2 (thousand dollars) at the beginning of each month. The amount of money he needs to spend during a month is independent of the amount he has and is equal to i with probability $...
0 votes
0 answers
30 views

How to use here Markov chain, to compute the required probability?

Suppose that balls are successively distributed among 8 urns, with each ball being equally likely to be put in any of these urns. What is the probability that there will be exactly 3 nonempty urns ...
0 votes
0 answers
12 views

sum of expected differences in time series

I have a (markov) decision process without reward. I am estimating the expected state differences $\mathbb{E}[\delta_t]$ where $\delta_t = X_{t+1} - X_t$. A state $X_{t+1}$ can be expressed as $X_0 + \...
1 vote
1 answer
50 views

Generate conditional probability matrix for simulation

I want to generate a categorical random variable conditioned on other variables, for example, generate $X$ from $Y$ by $P(X|Y)$, or generate $X$ from $Y_1,Y_2$ by $P(X|Y_1,Y_2)$. Here, $P(X|Y)$ ...
0 votes
0 answers
16 views

Reinforcement Learning MDP stochastic policies

I am struggling to get an understanding behind stochastic policy in a MDP I was reading this paper and in section 3.2 they say At each time step $t$, corresponding to extracting sentence number $t$, ...
0 votes
0 answers
13 views

How to define relative value functions for multichain MDPs

I have been studying the average reward MDP, and I found that in most references, the average reward criterion is defined as $\rho(s)=\lim_{N\to\infty} E_\pi[\frac{1}{N}\sum^{N-1}_{t=0}r_\pi(s_t)]$, ...
0 votes
0 answers
23 views

How to test for equality two finite state markov chains?

I have the transition probabilities of the two markov chains. I think that maybe I could apply the Kullback–Leibler divergence two the stationary distributions of these Markov chains (if they are ...
0 votes
0 answers
28 views

Data Sufficiency for Markov Model

I have a question regarding the Markov model. Suppose I have N number of data in a table and I constructed a simple Markov model with their state transitions as shown below. How do I know if I have &...
6 votes
2 answers
569 views

Can any Models be "Bagged"?

I have been learning about "bagging" (bootstrap aggregation) - supposedly, there are many types of statistical models can be bagged together. For example, CART Decision Trees can be "...
  • 6,048
1 vote
0 answers
24 views

Memorylessness by way of additional dimensions

This is a somewhat broad question that occurred to me regarding the nature of memorylessness. Namely: Is there utility in considering systems which are themselves not memoryless, but then expanding ...
1 vote
1 answer
26 views

Goodness of fit test for a transition density function of a Markov process

Suppose that you have one realization $x = \{x_n\}_{n = 1}^{N}$ of the stochastic process $X = \{X_n\}_{n = 1}^{N}$ with state space $\mathbb{R}$. Assume that the process is Markovian, time-...
  • 11
1 vote
0 answers
12 views

Coding assignment autograder: Checking if a data stream is generated by a particular Markov process

I teach an introductory programming course in which most assignments ship with a set of tests students can use locally on their machine to validate that their code works as intended. Many years ago we ...
0 votes
0 answers
16 views

continuous time markov chain sample size

Is that the same if I have one patient with one Markov chain of length 100, and 50 patients with 50 Markov chains of length 2? If they are the same, why? Thanks
  • 51
0 votes
1 answer
45 views

Balancing "Delayed Entry Bias" and "Survivorship Bias"?

This is a question I have always struggled with - suppose you have medical data on patients over a period of time. This includes information on how long they spent in different states: Admission, ...
  • 6,048
0 votes
1 answer
49 views

Can some Survival Models "Dominate" other Survival Models?

I recently heard an interesting interpretation of Survival Models : A "standard" Survival Analysis problem (e.g. where at the end of the study, observations can either be "Censored"...
  • 6,048
0 votes
0 answers
31 views

Showing that random process is stationary

Suppose i have $x_t, \bar{x_t}, t\in \mathbb{Z_+}$ independent 2-states $\{0, 1\}$ Markov chains with positive transition probabilities. Initial states are $x_0 = 0; \bar{x}_0 = 1$. For which positive ...
2 votes
1 answer
59 views

Why do we need to Define "Valid" State Transitions in a Multi-State Model?

I was watching this video (https://www.youtube.com/watch?v=Wy-WmY6x4tg) and the presenter mentions (@ 8:10) that in a Multi-State Model, the user is required to specify number of "States" ...
  • 6,048
4 votes
1 answer
119 views

Markov Chain Monte Carlo with known normalisation

I would like to compute the expectation value $\langle O \rangle = \sum_x P(x) O(x)$ of some random variable over an extremely large sample space that I cannot simply exhaustively go through. Usually ...
  • 141
1 vote
1 answer
41 views

Expectation of transition in a Markov process

I have the following transition matrix for a Discrete Markov process with 3 states, say states A, B and C: \begin{bmatrix}0.985992 & 0.0134092 & 0.000599272\\ 0.0265225 & ...
  • 123
1 vote
0 answers
28 views

How to map a sequence to a transition matrix

I have the following transition matrices, one for Maria and one for Anna: ...
  • 391
0 votes
0 answers
16 views

The gamble problem to find limiting stationary distribution

[This question was asked here before, but I did not get satisfactory response] I was trying to solve the following problem: I find out the transition matrix is: $$P = \begin{bmatrix} 1 & 0 &0 ...
0 votes
0 answers
12 views

How does detailed balance relate to (conditional) expectation?

Let $(\mathsf{X}, \mathcal{X})$ be a measurable space and $\pi$ be a probability distribution on it. Let $\mathrm{K}:\mathsf{X}\times\mathcal{X}\to[0, 1]$ be a Markov kernel. We say that $\mathrm{K}$ ...
  • 1,560
0 votes
0 answers
22 views

First order PDF of a stochastic process

I've started studying about stochastic processes and I need some help in this question. A random number generator is making numbers by this process: First number (X0) is a sample from Normal Standard ...
  • 1
1 vote
1 answer
75 views

What type of Markov Chain is a random walk of a Knight on a chessboard?

Assume we have the following chessboard and we have a knight that starts at the top left corner of the board. On every move the Knight chooses reachable square (i.e. a valid chess move a Knight can ...
  • 129
2 votes
0 answers
29 views

Hidden Markov Model observing sequences

I have been trying to understand Hidden Markov Models but I often find myself confused. I have discussed with my tutor for further help however, he is often rude and does not help and so I have ...
  • 143
0 votes
0 answers
75 views

Prove stationary distribution of the CTMC with Cut method

Consider a CTMC on state space $S$ with generator $G$. Prove that a distribution $π$ on $S$ is a stationary distribution of the CTMC if and only if for any “cut” (partition) ($A$, $A^c$ ) with $A ⊂ S$:...
  • 1
0 votes
0 answers
27 views

Identification of the transition probability of a time homogeneous MDP with subsampling

I am dealing with a MDP (or a temporal causal SEM) problem with missing observations. I want to know under what assumptions the transition probability can be identified from the observation. ...
0 votes
0 answers
30 views

Why the memoryless Markov Property is desirable

The memoryless Markov property says future predictions only depend on the current status. With longitudinal data, we have all the past data recorded. Why cannot we make use of all the info? Why should ...
  • 51

1
2 3 4 5
24