Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
1
vote
1answer
17 views

can the prior probability change with time?

I was reading Markov Models for sequence modeling and stuck with my understanding(hypothesis). In Baye's theorem, can the prior probabilities change with time? If the prior probability at t=3 is 0.05,...
0
votes
0answers
18 views

Inference and simulation on Markov Chain [on hold]

If I want to do inference for the next state on Markov Chain, do I need to find state staionary distribution first and then I do inference? Or state stationary distribution has nothing to do with ...
0
votes
0answers
16 views

Measure the distance between two probability transition matrices

I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
0answers
12 views

Markov Chain Monte Carlo for Decryption Purposes

I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
1
vote
1answer
16 views

What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?

I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
0
votes
0answers
6 views

What are the relationships among Markov Property, Stationarity, and Time Invariance

I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...
0
votes
0answers
19 views

How to train a single discrete-time markov model?

I have a training set of sequences. I want to reach a discrete time Markov model (transition probability matrix). Is there a Bayesian way other than MLE to achieve this?
0
votes
0answers
21 views

How to estimate Markov Chain probabilities from a given NCD dataset by using MLE in R [closed]

I have a dataset about no claim discount(NCD)car insurance scheme with 10,000 policyholders in the current policy year. the question is to estimate the probability transition matrix for the Markov ...
3
votes
1answer
50 views

Spread of number of steps to reach absorbing state in markov chain

I know how to calculate the variance of the number of steps in an absorbing markov chain. However, I am not sure that the distribution of the number of steps is normal. Therefore I would like to ...
-1
votes
0answers
19 views

Creating the input for mcl algorithm

I would like to implement mcl (Markov clustering algorithm) in a dataset like this one: ...
0
votes
0answers
15 views

How to find exact month of regime shifts in Markov switching model in R?

I'm looking into industry specific merger waves. I have used the Markov switching model (AR(0)) to identify periods with high M&A activity (merger wave periods) and periods with lower/normal M&...
0
votes
0answers
88 views

Understanding Convergence of Transitional Probabilities via Krylov Bogolivob Theorem

Could anyone break and write a few more lines for me of the proof of the Theorem 1.10 here, by Prof. Martin Hairer, on page 7 in particular. I am not understanding what he means by (1) the continuity ...
0
votes
0answers
33 views

Applicability of Markov Models for predicting user input

I am trying to predict user action based on the shown content of different modules. Lets assume the user sees a page where several content may be shown or not. The next action of the user depends on ...
1
vote
2answers
43 views

Time distibution in Markov chain

Let $E=\{A,B\}$ be a set and $X_{1,t}, X_{2,t}, X_{3,t}$ three independent Markov chains on the set $E$ with respective transition probability $P^{(1)}, P^{(2)}, P^{(3)}$ where $$P^{(i)}=\begin{...
2
votes
2answers
58 views

Proof of (weak) consistency for an unbiased estimator

I want to prove a theorem stating: An unbiased estimator $\hat{\theta}$ of the unknown parameter $\theta$ is consistent if $V(\hat{\theta}_n$) $\to0$ for ${n\to\infty}$. I've tried using the ...
0
votes
0answers
13 views

Data augmentation for animal measurements

I am looking for methods to do data augmentation for an animal study. What I want to do is generate a large number of data points by using a min and max measurements of the animal. So I have 2 ...
0
votes
0answers
4 views

MLE with higher order markov chain model

I am trying to estimate my Markov chain model. I understand that in a standard model, a higher order will lead to overfitting and higher likelihood. Hence, one can use AIC/BIC to find the order of the ...
0
votes
1answer
19 views

Hidden States in Hidden Markov Model

I am using Hidden Markov Models, having observations as continuous variable and states as discrete variable. I can use the observations to train HMM model and generate n number of states(say 2 hidden ...
1
vote
0answers
61 views

Does the Markov property always hold for a state-space structure?

Markov Property: $p({\bf x}_t | {\bf x}_1, \ldots, {\bf x}_{t-1}) = p({\bf x}_t | {\bf x}_{t-1})$ Consider the following model for which the hidden states are ${\bf x}_t$ and the observations are ${\...
1
vote
2answers
59 views

Irreducible markov chain problem

Let $P$ be a stationary transition probability matrix of the markov chain $ \{X_n , n \ge 0\}, $ which is irreducible and every state has a period 2. Further suppose that the markov chain $\{Y_n , n \...
1
vote
0answers
24 views

How to increase the total number of iterations it takes to converge a MDP?

I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge? For a given MDP which converges in 3 iterations, what setting ...
0
votes
1answer
14 views

Is this Markov Chain calculation correct?

$S=\{1,2\}$ $\alpha = (1/2, 1/2)$ $P= \begin{bmatrix} 1/2&1/2\\ 0&1\end{bmatrix} $ Find $P(X_1=1 | X_0=1)$ Given solution: $P(X_1=1 | X_0=1) = \frac{P(X_1=1 , X_0=1)}{P(...
0
votes
0answers
16 views

Programing a discrete time Markov chain simulation in R with individual-specific transition intensities

I need to simulate a discrete time Markov chain with individual-specific transition intensities (no differential equations) to model the prevalence of an infectious disease in an ageing population. I ...
0
votes
0answers
19 views

Markov chains and Equilibrium distribution

Two yachts, sailed by "Yacht 1" and "Yacht 2" respectively are sailing around a course. If the teams are even at the beginning of a lap then during that lap they embark on a duel and one team gains a ...
0
votes
0answers
16 views

Training Hidden Markov model with GMM, nan appears after some iterations

Problem During the training process of my continuous observation sequence data using HMM with GMM mixtures, the cost function reduces gradually and it becomes NaN ...
1
vote
0answers
27 views

conversion prediction / ROC curve using markov chains for channel attribution

I am currently working on a project on multi-channel attribution, using the channel attribution package from Altomare & Loris (2018), which uses markov chains for attribution. A walk-through of ...
1
vote
0answers
19 views

Geometric distribution with multiple success state and markovian succes probability

Let $X_t$ be a random process in the space $E:=\{F, S_1, S_2, S_3\}$ for each $t$. We can see it like it is a game where we can win in three different ways or we can fail. We play until we fail for ...
0
votes
0answers
15 views

Interrogating the results of the Markov simulation - Help and feedback highly appreciated

0 I have built a Markov chain with which I can simulate the daily routine of people (activity patterns). Each simulation day is divided into 144-time steps and the person can carry out one of ...
0
votes
2answers
37 views

understanding time-homogeneous markov chain

Could anyone help me understand the definition on page 7 definition 2.25 here? I do not understand the notation $(P(a))(A)$ - what does this mean? Also, is $P(a, A)$ a probability measure from the ...
2
votes
0answers
22 views

What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?

The expected average reward for a policy $\pi$ is: $$ \rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$ where $r_t$ is the reward obtained at time $t$ following policy $\pi$. ...
0
votes
0answers
12 views

Markov Chain for Different Groups of Time Series

I would like to use (hidden) Markov Chain to predict $X[t+1]$ stock price. Historical data for top biggest 500 companies for last 10 years will be used for training. The error would be sum of $...
2
votes
1answer
55 views

Metropolis-Hastings - interpreting the transition kernel: alpha*proposal

I thought I had great intuition and mathematical understanding of the Metropolis-Hastings algorithm, until closer inspection... as I started compiling my notes, I realized I do not understand the ...
1
vote
0answers
25 views

Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions

I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$. Now my idea is to run ...
0
votes
0answers
11 views

Zero Sum problem with MDP formulation and the difference with minimax approach

Suppose that we have formulate a zero sum game with MDP and Ua(s) and Ub(s) are the utilities of A and B in the s state. Suppose that all rewards and utilities are calculated from the A's point of ...
1
vote
1answer
32 views

How can I compute $P(X_1 = 1|X_0 = 1)$ from the given data?

I know how to mathematically calculate the probability of various Markov Properties. But, how can I calculate Markov probability from data? Suppose, I have a Markov Chain as follows: $S=\{1, 2\}$ ...
2
votes
0answers
32 views

Understanding Approximate Dynamic Programming

I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
1
vote
1answer
29 views

Markov Blanket of two nodes?

I'm trying to solve a question and it has asked for (I feel like I'm confused by everything, would apprecaite some help, thanks), the Markov blanket of {c,d}. From what I've read so far, Markov ...
0
votes
1answer
25 views

MCMC changing multiple parameters

What is the standard way of drawing multiple parameters for a MCMC run? Say I have 9 parameters, what is the most efficient method to get all parameters to explore their distributions properly?
1
vote
1answer
40 views

What happens if the observations are connected in a hidden Markov model (HMM)?

Suppose that we have an HMM with hidden variables $X_t$ and observed variables $Y_t$. Why do we always assume $p(Y_t|X_t)$? What happens if we have $p(Y_t|X_t, Y_{t-1})$? Is it because that wouldn't ...
2
votes
1answer
106 views

Manual simulation of Markov Chain in R

Consider the Markov chain with state space S = {1, 2}, transition matrix and initial distribution α = (1/2, 1/2). Simulate 5 steps of the Markov chain (that is, simulate X0, X1, . . ...
0
votes
1answer
32 views

Hidden Markov Model probability of producing a sequence

Suppose that we have two models for a 2-state HMM and both have two output symbols: $A$ and $B$. Model 1: Transition probabilities: $a_{11}=0.6$, $a_{12}=0.4$, $a_{21}=0.0$, $𝑎_{22}=1.0$. Output ...
1
vote
1answer
57 views

For a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?

I am new to Markov Chains and using this concept in statistics. For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$? If yes, how can I prove that?
0
votes
1answer
44 views

limit and stationary distribution of a Markov chain

Consider a Markov chain on the non-negative integers with transition probabilities 􏰀$1/2$ if $y=x+1$ and $1/2$ if $y=0$. Find $\lim_{n \to \infty} P(X_{n}=0)$. Is this limit the same as the ...
0
votes
0answers
12 views

Countably infinite state space for Continuous Time Markov Chains

I have been unsuccessfully browsing the internet for resources related to Continuous Time Markov Chains with a countably infinite state space, for example $\mathbb{N}$. Is there any developed theory ...
4
votes
2answers
89 views

Why does Judea Pearl call his causal graphs Markovian?

In his texts on causality, Judea Pearl always refers to the simplest graphs he uses, i.e. the acyclic graphs with independent confounders, as Markovian. I don't see why these graphs contain anything ...
0
votes
1answer
28 views

Compute the Limiting Distribution

Consider the transition matrix $ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $ for general $2$-state Markov Chain $(0 \le p, q\le 1)$. Find the limiting distribution ...
0
votes
0answers
12 views

Model selection with this model of a large number of components

I have a discrete time Markov Chain $\{X_n: n \in \mathbb{N}_0\}$ with unknown transition matrix $P \in \mathbb{R}^{M \times M}$ on the state space $\mathcal{S}_X = \{1,2, \dots, M\}$, with $M \geq 2$....
1
vote
1answer
23 views

How can I compute expected return time of a state in a Markov Chain?

I was watching a YouTube video regarding the calculation of expected return time of a Markov Chain. https://www.youtube.com/watch?v=X_Ll0-Ytu7U&vl=en I haven't understood the calculation of $m_{...
0
votes
0answers
17 views

Understanding the meaning of transition rates in a CTMC

I am reading the queueing theory volume 1 by Kleinrock. In the chapter on Continuous Time Markov Chain(CTMC), the author defines the infinitesimal generator, $Q(t)$ as having the following elements: $...
1
vote
0answers
30 views

Markov property of manifest variables in longitudinal ESM

I am working on a longitudinal ESM model were the indicators are (highly) autocorrelated. This means that the classic cross-lagged models of panel data analysis cannot be used directly. I have ...