Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
924
questions
1
vote
1answer
17 views
can the prior probability change with time?
I was reading Markov Models for sequence modeling and stuck with my understanding(hypothesis).
In Baye's theorem, can the prior probabilities change with time? If the prior probability at t=3 is 0.05,...
0
votes
0answers
18 views
Inference and simulation on Markov Chain [on hold]
If I want to do inference for the next state on Markov Chain, do I need to find state staionary distribution first and then I do inference? Or state stationary distribution has nothing to do with ...
0
votes
0answers
16 views
Measure the distance between two probability transition matrices
I have a probability transition matrix $P$ that contains some values very close to zero. I want to sparsify this matrix by taking the k largest values for each row and setting the others to zero. For ...
0
votes
0answers
12 views
Markov Chain Monte Carlo for Decryption Purposes
I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
1
vote
1answer
16 views
What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?
I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
0
votes
0answers
6 views
What are the relationships among Markov Property, Stationarity, and Time Invariance
I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...
0
votes
0answers
19 views
How to train a single discrete-time markov model?
I have a training set of sequences. I want to reach a discrete time Markov model (transition probability matrix).
Is there a Bayesian way other than MLE to achieve this?
0
votes
0answers
21 views
How to estimate Markov Chain probabilities from a given NCD dataset by using MLE in R [closed]
I have a dataset about no claim discount(NCD)car insurance scheme with 10,000 policyholders in the current policy year. the question is to estimate the probability transition matrix for the Markov ...
3
votes
1answer
50 views
Spread of number of steps to reach absorbing state in markov chain
I know how to calculate the variance of the number of steps in an absorbing markov chain. However, I am not sure that the distribution of the number of steps is normal. Therefore I would like to ...
-1
votes
0answers
19 views
Creating the input for mcl algorithm
I would like to implement mcl (Markov clustering algorithm) in a dataset like this one:
...
0
votes
0answers
15 views
How to find exact month of regime shifts in Markov switching model in R?
I'm looking into industry specific merger waves. I have used the Markov switching model (AR(0)) to identify periods with high M&A activity (merger wave periods) and periods with lower/normal M&...
0
votes
0answers
88 views
Understanding Convergence of Transitional Probabilities via Krylov Bogolivob Theorem
Could anyone break and write a few more lines for me of the proof of the Theorem 1.10 here, by Prof. Martin Hairer, on page 7 in particular. I am not understanding what he means by (1) the continuity ...
0
votes
0answers
33 views
Applicability of Markov Models for predicting user input
I am trying to predict user action based on the shown content of different modules. Lets assume the user sees a page where several content may be shown or not. The next action of the user depends on ...
1
vote
2answers
43 views
Time distibution in Markov chain
Let $E=\{A,B\}$ be a set and $X_{1,t}, X_{2,t}, X_{3,t}$ three independent Markov chains on the set $E$ with respective transition probability $P^{(1)}, P^{(2)}, P^{(3)}$ where $$P^{(i)}=\begin{...
2
votes
2answers
58 views
Proof of (weak) consistency for an unbiased estimator
I want to prove a theorem stating:
An unbiased estimator $\hat{\theta}$ of the unknown parameter $\theta$ is consistent if $V(\hat{\theta}_n$) $\to0$ for ${n\to\infty}$.
I've tried using the ...
0
votes
0answers
13 views
Data augmentation for animal measurements
I am looking for methods to do data augmentation for an animal study. What I want to do is generate a large number of data points by using a min and max measurements of the animal.
So I have 2 ...
0
votes
0answers
4 views
MLE with higher order markov chain model
I am trying to estimate my Markov chain model. I understand that in a standard model, a higher order will lead to overfitting and higher likelihood. Hence, one can use AIC/BIC to find the order of the ...
0
votes
1answer
19 views
Hidden States in Hidden Markov Model
I am using Hidden Markov Models, having observations as continuous variable and states as discrete variable. I can use the observations to train HMM model and generate n number of states(say 2 hidden ...
1
vote
0answers
61 views
Does the Markov property always hold for a state-space structure?
Markov Property: $p({\bf x}_t | {\bf x}_1, \ldots, {\bf x}_{t-1}) = p({\bf x}_t | {\bf x}_{t-1})$
Consider the following model for which the hidden states are ${\bf x}_t$ and the observations are ${\...
1
vote
2answers
59 views
Irreducible markov chain problem
Let $P$ be a stationary transition probability matrix of the markov chain $ \{X_n , n \ge 0\}, $ which is irreducible and every state has a period 2. Further suppose that the markov chain $\{Y_n , n \...
1
vote
0answers
24 views
How to increase the total number of iterations it takes to converge a MDP?
I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge?
For a given MDP which converges in 3 iterations, what setting ...
0
votes
1answer
14 views
Is this Markov Chain calculation correct?
$S=\{1,2\}$
$\alpha = (1/2, 1/2)$
$P= \begin{bmatrix}
1/2&1/2\\
0&1\end{bmatrix} $
Find
$P(X_1=1 | X_0=1)$
Given solution:
$P(X_1=1 | X_0=1) = \frac{P(X_1=1 , X_0=1)}{P(...
0
votes
0answers
16 views
Programing a discrete time Markov chain simulation in R with individual-specific transition intensities
I need to simulate a discrete time Markov chain with individual-specific transition intensities (no differential equations) to model the prevalence of an infectious disease in an ageing population. I ...
0
votes
0answers
19 views
Markov chains and Equilibrium distribution
Two yachts, sailed by "Yacht 1" and "Yacht 2" respectively are sailing around a course. If the teams are even at the beginning of a lap then during that lap they embark on a duel and one team gains a ...
0
votes
0answers
16 views
Training Hidden Markov model with GMM, nan appears after some iterations
Problem
During the training process of my continuous observation sequence data using HMM with GMM mixtures, the cost function reduces gradually and it becomes NaN ...
1
vote
0answers
27 views
conversion prediction / ROC curve using markov chains for channel attribution
I am currently working on a project on multi-channel attribution, using the channel attribution package from Altomare & Loris (2018), which uses markov chains for attribution. A walk-through of ...
1
vote
0answers
19 views
Geometric distribution with multiple success state and markovian succes probability
Let $X_t$ be a random process in the space $E:=\{F, S_1, S_2, S_3\}$ for each $t$. We can see it like it is a game where we can win in three different ways or we can fail. We play until we fail for ...
0
votes
0answers
15 views
Interrogating the results of the Markov simulation - Help and feedback highly appreciated
0
I have built a Markov chain with which I can simulate the daily routine of people (activity patterns). Each simulation day is divided into 144-time steps and the person can carry out one of ...
0
votes
2answers
37 views
understanding time-homogeneous markov chain
Could anyone help me understand the definition on page 7 definition 2.25 here? I do not understand the notation $(P(a))(A)$ - what does this mean?
Also, is $P(a, A)$ a probability measure from the ...
2
votes
0answers
22 views
What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?
The expected average reward for a policy $\pi$ is:
$$ \rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$
where $r_t$ is the reward obtained at time $t$ following policy $\pi$.
...
0
votes
0answers
12 views
Markov Chain for Different Groups of Time Series
I would like to use (hidden) Markov Chain to predict $X[t+1]$ stock price. Historical data for top biggest 500 companies for last 10 years will be used for training.
The error would be sum of $...
2
votes
1answer
55 views
Metropolis-Hastings - interpreting the transition kernel: alpha*proposal
I thought I had great intuition and mathematical understanding of the Metropolis-Hastings algorithm, until closer inspection... as I started compiling my notes, I realized I do not understand the ...
1
vote
0answers
25 views
Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions
I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$.
Now my idea is to run ...
0
votes
0answers
11 views
Zero Sum problem with MDP formulation and the difference with minimax approach
Suppose that we have formulate a zero sum game with MDP and Ua(s) and Ub(s) are the utilities of A and B in the s state.
Suppose that all rewards and utilities are calculated from the A's point of ...
1
vote
1answer
32 views
How can I compute $P(X_1 = 1|X_0 = 1)$ from the given data?
I know how to mathematically calculate the probability of various Markov Properties.
But, how can I calculate Markov probability from data?
Suppose, I have a Markov Chain as follows:
$S=\{1, 2\}$
...
2
votes
0answers
32 views
Understanding Approximate Dynamic Programming
I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
1
vote
1answer
29 views
Markov Blanket of two nodes?
I'm trying to solve a question and it has asked for (I feel like I'm confused by everything, would apprecaite some help, thanks),
the Markov blanket of {c,d}. From what I've read so far, Markov ...
0
votes
1answer
25 views
MCMC changing multiple parameters
What is the standard way of drawing multiple parameters for a MCMC run? Say I have 9 parameters, what is the most efficient method to get all parameters to explore their distributions properly?
1
vote
1answer
40 views
What happens if the observations are connected in a hidden Markov model (HMM)?
Suppose that we have an HMM with hidden variables $X_t$ and observed variables $Y_t$. Why do we always assume $p(Y_t|X_t)$? What happens if we have $p(Y_t|X_t, Y_{t-1})$? Is it because that wouldn't ...
2
votes
1answer
106 views
Manual simulation of Markov Chain in R
Consider the Markov chain with state space S = {1, 2}, transition matrix
and initial distribution α = (1/2, 1/2).
Simulate 5 steps of the Markov chain (that is, simulate X0, X1, . . ...
0
votes
1answer
32 views
Hidden Markov Model probability of producing a sequence
Suppose that we have two models for a 2-state HMM and both have two output symbols: $A$ and $B$.
Model 1:
Transition probabilities: $a_{11}=0.6$, $a_{12}=0.4$, $a_{21}=0.0$, $𝑎_{22}=1.0$.
Output ...
1
vote
1answer
57 views
For a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?
I am new to Markov Chains and using this concept in statistics.
For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?
If yes, how can I prove that?
0
votes
1answer
44 views
limit and stationary distribution of a Markov chain
Consider a Markov chain on the non-negative integers with transition
probabilities $1/2$ if $y=x+1$ and $1/2$ if $y=0$. Find $\lim_{n \to \infty} P(X_{n}=0)$.
Is this limit the same as the ...
0
votes
0answers
12 views
Countably infinite state space for Continuous Time Markov Chains
I have been unsuccessfully browsing the internet for resources related to Continuous Time Markov Chains with a countably infinite state space, for example $\mathbb{N}$.
Is there any developed theory ...
4
votes
2answers
89 views
Why does Judea Pearl call his causal graphs Markovian?
In his texts on causality, Judea Pearl always refers to the simplest graphs he uses, i.e. the acyclic graphs with independent confounders, as Markovian. I don't see why these graphs contain anything ...
0
votes
1answer
28 views
Compute the Limiting Distribution
Consider the transition matrix
$ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $
for general $2$-state Markov Chain $(0 \le p, q\le 1)$.
Find the limiting distribution ...
0
votes
0answers
12 views
Model selection with this model of a large number of components
I have a discrete time Markov Chain $\{X_n: n \in \mathbb{N}_0\}$ with unknown transition matrix $P \in \mathbb{R}^{M \times M}$ on the state space $\mathcal{S}_X = \{1,2, \dots, M\}$, with $M \geq 2$....
1
vote
1answer
23 views
How can I compute expected return time of a state in a Markov Chain?
I was watching a YouTube video regarding the calculation of expected return time of a Markov Chain.
https://www.youtube.com/watch?v=X_Ll0-Ytu7U&vl=en
I haven't understood the calculation of $m_{...
0
votes
0answers
17 views
Understanding the meaning of transition rates in a CTMC
I am reading the queueing theory volume 1 by Kleinrock. In the chapter on Continuous Time Markov Chain(CTMC), the author defines the infinitesimal generator, $Q(t)$ as having the following elements:
$...
1
vote
0answers
30 views
Markov property of manifest variables in longitudinal ESM
I am working on a longitudinal ESM model were the indicators are (highly) autocorrelated. This means that the classic cross-lagged models of panel data analysis cannot be used directly. I have ...
