Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,187
questions
0
votes
0answers
4 views
Using Markov Chains to predict market share for prescription drugs
I wish to understand if the approach I am researching holds up to analytical rigor.
I am planning to use historical data of 4 prescription drugs to forecast the market share of the same 4 drugs some ...
0
votes
0answers
15 views
Probability of simultaneous events given different conditions [closed]
We are dealing with a probability-theoretic back-of-the-envelope calculation for a thesis and have run into a problem - hope somebody can help!
Our case is the following:
We observe an agent, $i$, ...
0
votes
0answers
15 views
How to produce a transitional matrix when there is no example of some starting states in your data using R?
I am working toward building a Markov chain model, and need to produce a transition matrix for the model to be built.
Using three categorical variables, Student Type, Full-time/Part-Time status, and ...
1
vote
0answers
10 views
LR test for overlapping return data using bootstrap
I wish to test a null hypothesis as in Christoffersen (1998) to see whether a sequence of Value-at-Risk forecasts $Q_t(p) \in \mathcal{F}_t$ possesses correct conditional coverage. Here $p \in [0,1]$ ...
0
votes
0answers
9 views
derivation for piecewise deterministic Markov process
I was reading a formulation of piecewise-deterministic Markov process $\Pi_t$, $ t \in \mathbb{R}_+$. In particular $\Pi_t$ is defined as $\Pi_t = P (X_t | \mathcal{F}_{\lfloor t/ \Delta \rfloor}) $ ...
0
votes
0answers
13 views
Deciding on sequence prediction approach
I have a space of 128 states expressed with binary bitstrings. The transition probabilities among these states are known to me, as follows:
...
0
votes
0answers
19 views
Event-based markov chain with observed covariates at non-constant time intervals
I am working on a project where I need to identify the traffic state at intersections. More specifically, I want to classify the situation in 5 states: undersaturation (where the traffic needs to stop ...
0
votes
0answers
19 views
When does a stationary distribution for a CTMC not exist?
Recalling that a Continuous-time Markov Chain (CTMC) is defined by its generator matrix $Q$, when does a stationary distribution for a CTMC not exist? Or when does there not exist a probability vector ...
1
vote
0answers
28 views
Markov Chain for Amino Acid Sequence
I was wondering if it is possible to apply Markov Chain to this amino acid sequence "ddvlsldeddddsdyncgednd" to find the probability of this sequence forming in this particular positional ...
0
votes
0answers
14 views
Proving positive recurrence for a finite state space DTMC
Let $$P =
\begin{pmatrix}
0.3 & 0.5 & 0 & 0 & 0 & 0.2 \\
0 & 0.5 & 0 & 0.5 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 & 0 \\
0 & 0.3 & 0 & 0 &...
0
votes
2answers
71 views
Markov Chains - Can we set the Initial distribution as $X_{1}$ instead of $X_{0}$ to calculate a non-conditional probability?
Had a small question when I was going through Ross – Probability Models. Given a discrete-time Markov chain
$\{X_{n}: n=1,2,3 \ldots\}$ with a state space $\mathcal{S} = \{0,1\}$, and an initial ...
0
votes
0answers
10 views
Rankings transition matrix
I have heard of ratings transition matrices in finance, where it is possible to derive the probability of moving from one rating to another in a Markov chain process. If I have N individuals who are ...
1
vote
1answer
21 views
What does it mean for a Markov CHAIN to be recurrent (not just a state)?
There are many resources offering equivalent definitions of recurrence for a state in a Markov Chain - for example, state $x$ is recurrent if, starting in state $x$ you will eventually return to state ...
0
votes
1answer
22 views
Question regarding limiting probability
Does there exist an irreducible discrete time Markov chain such that all limiting probabilities exist, i.e. $\lim\limits_{k \to \infty} \mathbb{P}(X_k = i)$ exists for all states $i$, and $\sum_{i} \...
0
votes
0answers
29 views
How to calculate var(var) in time series
I try to calculate var(var) with Markov Process in time series by Bartlett(1946, P28) https://www.jstor.org/stable/2983611?seq=2#metadata_info_tab_contents , I use quadratic form calculate,but I didn'...
0
votes
0answers
8 views
When to use a markov switching model?
When can you use a markov switching model? For example if I was looking at the effect of inflation on bank liquidity, could it be used here?
0
votes
0answers
4 views
Question on the transition analysis and if they happen by chance (R)
I am wondering if somebody could suggest there I can find a tutorial or a book how to write the code in for following problem.
As the title sad I have a transition matrix , and I need to know id the ...
0
votes
0answers
13 views
Local independence vs global independence in markov network
I am having a hard time understanding the basic differences between the local independence and global independence of a markov network. Please help me illustrate with a graph or any example
1
vote
1answer
25 views
Markov Chains examples
So I've been given four examples of stochastic processes, and asked which are Markov chains:
The four examples are henceforth:
"Keep rolling a die and let $X_n$ be the value of the n-th roll.&...
0
votes
0answers
11 views
calculating factors of markov chain [closed]
How do i calculate these numbers(image below) in a markov chain. What is the intuition behind this.
I understand, since these are undirected graphs we are trying to calculate the strength of edges as ...
0
votes
0answers
10 views
From two-state Markov Chains to Mean Time Between Failures / To Repair
Let $\mathcal T = \{1,2,\ldots,T\}$ denote the set of points in time, $S = \{0,1\}$ the state space, $X = (X_t)_{t \in \mathcal T} \in S^\mathcal T$ a time series, $\alpha = \mathbb P(X_{t+1} = 0 \mid ...
1
vote
0answers
12 views
“Forcing” equal probabilities in the matrix exponential of a Markov intensity matrix
I have an upper-right triangular transition intensity matrix $Q$ for a 7-state Markov model (with states $X_1,X_2,...,X_7$), from which I numerically calculate the matrix exponential to derive a ...
1
vote
1answer
42 views
Is the data-processing inequality still true for higher-order Markov chains?
The data processing inequality states that if $X_1,X_2,...,X_n$ form the first-order Markov chain:
$$
X_1 \rightarrow X_2 \rightarrow \cdots \rightarrow X_n
$$
Then for all $i \leq j \leq k \leq l$:
$$...
0
votes
0answers
19 views
Repeated Poisson draws - stationarity
Many individuals (say, $N=500$ or more) independantly draw from a Poisson distribution with mean $\lambda$. This gives the "Time until next update", $t_u$, for each respective individual.
At ...
1
vote
0answers
11 views
Forward equations and global balance equations for SIRS model?
I have a system with 3 states: 0, 1, 2 (disease-free, infectious and isolated - is this a SIRS model? It's looking at an individual person rather than a population). Transition from 0 to 1 takes place ...
3
votes
1answer
31 views
Explain formally how expected time to hit 0 from two is the sum of the expected time to hit 1 from 2 and 0 from 1
I have a symmetric random walk on the integers with probability $p$ and $q$ of going up and down respectively started at $X_0 = 2$.
Let
$$
T^0 = \min\{ n > 0: X_n = 0\}, T^1 = \min\{ n > 0: X_n =...
1
vote
0answers
17 views
Creating transition states for a second order Markov Chain for Attribution
I'm pretty new to Markov Chains, and I'm exploring them for attribution. I'd like to play around with higher orders but I'm a little confused on how to do this.
Most resources I can find show a second ...
0
votes
0answers
28 views
Prove the given transformation of a Markov Chain is a Markov Chain
Let $\{X_n:n=0,1,\dots\}$ be a discrete time Markov chain on state space $S$. We define $Y_n$ as the following:
$$Y_n=(X_n, X_{n+1})$$
Prove that $\{Y_n:n=0,1,\dots\}$ is a discrete time Markov chain
...
0
votes
0answers
16 views
Is it possible to eye a reversible Markov Chain by it's transition matrix?
I was wondering if it's possible. Many of the other Markov-Chain properties can be (somewhat) easily eyed from the transition matrix. (e.g. irreducible - if there's an absorbing state (1 in the ...
4
votes
1answer
29 views
Calculating the expected number of visits to a state by a DTMC
Suppose we have a DTMC $X$ : $\{X_n : n = 0, 1,2,\dots\}$, a transition probability matrix $P$, and state space $S = \{1,2,3\}$. Suppose I want to calculate the expected amount of times we visit state ...
1
vote
0answers
42 views
Using Forward Backward algorithm to find posterior probability of all possible states
I understand that Viterbi finds the most probable sequence of states.
However, I want the probability of all possible sequences of states.
I understand that FB algorithm can be used to find the ...
0
votes
0answers
14 views
Why is Markov Property useful for modeling systems?
I came across this question in my data science interview practice, and am not sure how to answer. Markov models are based on the assumption of Markov property, so it is useful for modeling Markov ...
3
votes
1answer
20 views
Conditional probabilities with Discrete Time Markov Chain non-perishable inventory
Suppose $\{X_n : n =0,1,2,\dots \}$ is a DTMC that represents the inventory level at the end of day $n$. We have inventory policy $(2,4)$, i.e., if $X_n < 2$, we order enough units to have ...
4
votes
1answer
107 views
Is gamma actually an efficient way to weigh future rewards in reinforcement learning?
Typically the discounted sum of rewards is defined as follows:
G_t = Sum(gamma ** n * reward_t...)
But this means that rewards are worth exponentially less with ...
0
votes
0answers
24 views
MCMC beginner question at an example chain plot: Do I need more steps? How much burn-in do I need, if I can tell already?
I am using the emcee python library to fit a model to data via MCMC. Below an example plot for the chain of one of my parameters. Here I ran 1000 steps with 100 walkers.
Now I have two beginner ...
0
votes
0answers
12 views
Fitting the transition matrix of a Markov chain given the series of state
Almost all examples I can found on Markov chain and python are giving the transition matrix as known. Is there any library that can fit the transition matrix from the series of state observations?
1
vote
0answers
41 views
Prove the joint distribution of an AR(1) process is Multivariate Gaussian
I'm struggling a bit with the proof of this if anyone can help!
I keep ending up going in circles with the conditional probabilities and I don't know what are the right steps to take.
For an $AR(1)$ ...
2
votes
1answer
26 views
To compute the class of states for the given transition probability matrix
I have been given the following transition probability matrix of a markov chain:
$P = \begin{pmatrix}
\frac{3}{4}{} & 0 & \frac{1}{4} &0 \\
\frac{1}{2} & 0 & 0 & \frac{1}{2}\\ ...
1
vote
1answer
120 views
A Markov chain {Xn, n ≥ 0} with states 1, 2,3 has the transition probability matrix with an initial distribution (1/2,0,1/2), what is P(X1=3|X2=1)
A Markov chain {Xn, n ≥ 0} with states 1, 2,3 has the transition probability matrix P
\begin{bmatrix}0&0.4&0.6\\1&0&0\\0.3&0.3&0.4\end{bmatrix}
with an initial distribution A (...
0
votes
1answer
36 views
Whats exactly deterministic and non deterministic in deterministic and nondeterministic MDP policies?
Consider below Markov Decision Process:
Blue hexagons are states and orange circles are actions. I have rather simple confusion.
What will be nature of deterministic and non deterministic MDPs? This ...
1
vote
1answer
69 views
Equivalence of two state Markov chain and sampling via geometric distribution
Let $\mathcal T = \{1,2,\ldots,T\}$ denote the set of points in time, $S = \{0,1\}$ the state space, $X = (X_t)_{t \in \mathcal T} \in S^\mathcal T$ a time series, $\alpha = \mathbb P(X_{t+1} = 0 \mid ...
0
votes
0answers
11 views
Is it possible to occupy two states at the same time in Markov process/chain?
As in the topic - is it possible for an agent in a system of Markov's fashion to move into two states at once?
0
votes
0answers
28 views
How to find the n-step matrix more efficiently?
Is there any clever way to find the n-step matrix of a chain? I have the following transition matrix
However, $p^{(n)}_{1,2}$ I spent a lot of time trying to find a way of recurrence along the paths ...
0
votes
0answers
20 views
Are invariant and stationary distribution the same thing?
I am reading a material about Markov chains and in it the author works on the Markov chains part discrete the invariant distribution of the process. However, when addressing the part of continuous ...
0
votes
0answers
20 views
Birth and death process and Counting processes - What is the relationship?
Well, I started to study the birth and death process and can I say that a counting process is an example of a pure birth process or that the birth and death process is an example of a counting process?...
0
votes
0answers
13 views
Markov process of order one a Martingale?
I have two questions, and I am very confused about the concepts
Can a Markov process of order one also be a Martingale?
Is any Markov process of order one also a Martingale?
can anyone help me solve ...
0
votes
0answers
23 views
Likelihood ratio test for Markov orders - What succession of tests?
I want to estimate the Markov order of a binary sequence. For that I calculated transition matrices and the log likelihoods for the orders of interest 0,1,2 and 3. Essentially the one with the highest ...
0
votes
0answers
9 views
Calculating marginal distribution of Markov process
i am studying markov processes and we have an example of a VAR process. i am trying to understand how to look at the marginal distribution so i can find a gaussian distribution for $Y_t$ which equates ...
0
votes
0answers
8 views
Queuing theory. Modification in the M/M/1 model
In a queue of type M/M/1 when making the following modification: when there are 3 customers in the system (2 in the queue and 1 being served) if another one arrives he will leave and never come back. ...
0
votes
0answers
9 views
Periodic Markov chain in 3D state space
Does someone have an example for a discrete time (time-homogeneous) Markov chain in a three-dimensional state space that is characterized by a transition matrix resulting in periodic behaviour?
I am ...
