Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,109
questions
0
votes
0answers
7 views
Strategic Multi Armed Bandit
As a part of my project, I have been tasked with formulating a multi-armed bandit problem with strategic arms. What I have found out is a Gittin's index approach to the problem provides a solution ...
0
votes
0answers
20 views
How to test that a discrete time series has markovian properties? How to use the Wilcoxon test to do so?
Let us assume I have a time series made of the following observations:
ts = c(163,18,53,189, 243, 101, 150, 39, 60,96,36,76,71,67,56,3,72,96,15,19)
How can I use the Wilcoxon test to determine if it ...
8
votes
3answers
563 views
Equilibrium distribution of Markov chain
The transition matrix is $$P =\begin{bmatrix} \frac12 & \frac12 & 0 & 0 \\ \frac12 & \frac12 & 0 & 0 \\ 0 & 0 & \frac13 & \frac23 \\ 0 & 0 & \frac13 & \...
2
votes
0answers
63 views
Long term expected win rate
I have the following game, you have some initial probability $p_i$ to win, every time you try and lose the probability increases by a constant $p_a$ and is reset to $p_i$ when you win.
For example if ...
1
vote
0answers
11 views
Gibbs sampling Bayesian conditional distribution for mean of a Normal distribution
first post here in CV. I'm currently working on a textbook exercise on Gibbs Sampling and got stuck on naming the distribution for one of the conditional distributions.
Question
Consider a normal ...
0
votes
0answers
25 views
Gambling game: If he wins he quit if he loses doubles the bet [closed]
A player is gambling based on the following strategy. The player bets $\$1$ at first round. If he
wins, he ends the game. If he loses, he doubled the bets to $\$2$ at the second round. If he wins,
he ...
2
votes
1answer
122 views
Markov chain - ergodic theorem
I need help with transition matrix of the following problem
On each evening, the owner of a car washes it with probability 0.6. Independently of that event, every night a dirty rain pours on the car ...
0
votes
0answers
27 views
Convergence of policy iteration with discount $\gamma=1$
This question is related to my question (and my comment of RobPratt's answer) at https://math.stackexchange.com/questions/3860303/markov-decision-process-with-target-states-and-shortest-path-as-only-...
0
votes
0answers
14 views
PDF of a two state Markov chain with binned observation
I am trying to come up with a PDF to describe experimental data, which I can describe well with a simple Monte Carlo simulation.
I have a two state Markov chain with equal transition probabilities ...
1
vote
0answers
31 views
expected number of dependent trials until n successes
In case of bernoulli experiments, I know that the expected value for the number of trials needed to have $n$ successes is n/p. (where p is probability of success for each trial). Now my question is, ...
36
votes
2answers
6k views
A fair die is rolled 1,000 times. What is the probability of rolling the same number 5 times in a row?
A fair die is rolled 1,000 times. What is the probability of rolling the same number 5 times in a row? How do you solve this type of question for variable number of throws and number of repeats?
2
votes
1answer
25 views
Predicting Graph Edge Connections
I have a set of nodes in 3d physical space. Some of those nodes are connected to one another by a graph edge, while others are not. Just because two nodes are physically close doesn't necessarily mean ...
1
vote
1answer
54 views
How to determine if a -long- time series made of discrete observations is markovian?
Let us assume I have a time series made of the following observations:
ts = c(163,18,53,189, 243, 101, 150, 39, 60,96,36,76,71,67,56,3,72,96,15,19)
How can I determine if it respects the Markov ...
0
votes
1answer
35 views
Calculation of transition probabilities of Markov Chain
I have just started learning markov chains and I need help on the following question:
Alice and Bob vote in each parliamentary election. If, in a certain election, Alice and Bob vote for the same ...
0
votes
0answers
3 views
Element by element t-test between two dependent Markov transition matrices
I want to conduct t-tests between each of the probabilities of an unconditional matrix and a conditional matrix generated from discrete-time Markov chains. The unconditional probabilities are derived ...
2
votes
1answer
35 views
Is a policy $\pi(s)$ on Markov decision process a random variable?
Citing Wikipedia:
The goal in a Markov decision process is to find a good "policy" for
the decision maker: a function $\pi$ that specifies the action
$\pi(s)$ that the decision maker will ...
0
votes
0answers
7 views
Statistically compare transition probabilities from two transition matrices
I have a dataset that is stratified by gender. I have fit a first-order Markov chain to both strata, giving transition matrices $M_1$ and $M_2$. Now I want to statistically test, for given $i, j$, ...
2
votes
1answer
34 views
How to solve a Markov Decision Problem with State Transition Matrix and Reward Matrix
I'm stuck in solving a simple dynamic probabilistic model. I have Three states {Sunny, Cloudy, Rainy}.
I have the ...
0
votes
0answers
19 views
state transition classification on terminal state
see https://datascience.stackexchange.com/questions/82169/state-transition-classification-on-terminal-state
0
I have data on a unit š which enters an entry state š0. This unit has some covariates š„ļæ½...
1
vote
0answers
48 views
Estimating conditional probabilities in the context of finite state discrete time Markov chains of order p
Consider a Markov chain with evenly observed discrete times, where the order of the Markov chain is 1. Then we may estimate the transition probability of the Markov chain using sample proportions. ...
1
vote
1answer
17 views
Does it make sense to make a conditional Markovianity assumption?
Please do excuse if the title is somewhat vague. As we know, a discrete-time Markov chain is a sequence of random variables $S_1,S_2,\cdots$ with the Markov property - i.e.
\begin{equation}
P[S_{t+1}=...
2
votes
1answer
31 views
How can I test if a time-series data satisfies Markov's property and it is a martingale?
My question is about investigating some properties of time-series.
How can I test if my time-series data satisfies Markov's property?
How can I test if my time-series data is a martingale?
I wonder ...
0
votes
1answer
18 views
Yule (preferential attachment process vs. birth&death process
What is the distinction between the Yule (preferential attachment) process and the birth&death process? Are they the same thing, called differently in different contexts, or is the birth&death ...
0
votes
0answers
13 views
How to āde-meanā a series so it can be used in MSGARCH R package?
0
I would like to use MSGARCH package to fit a Markov switching GARCH model. I know that there is a useful MSGARCH package in R but it does not allow to add a conditional mean equation like in rugarch ...
1
vote
0answers
40 views
How to calculate Markov Chain parameters knowing AR(1) coefficient $\rho$?
Given that a discrete-time two-states (low and high) Markov chain $X_t$ can take two values:
\begin{pmatrix}
X_l\\
X_h
\end{pmatrix}
with its transition probability matrix $P^{T}$:
\begin{pmatrix}
p_1 ...
2
votes
0answers
66 views
Carré du champ operator is a quadratic variation
Let $X_t$ be a real valued Markov process (starting at $x$) with generator $L$. Let $\Gamma(f)$ denote CarrƩ du champ operator i.e. $L(f^2) - 2f \cdot L (f)$. As far as I know under suitable ...
1
vote
1answer
26 views
Markov Switching Model with Markov trend
I have a time series with nonstationary data. Looking at the plot of the data it is clear that there are structural breaks. As data follow the stochastic process, I want to build Markov Switching ...
2
votes
0answers
18 views
How can I cluster sequential data?
Suppose that I have a sequence of vectors $y_n \in \mathbb{R}^m$ for $n \in \{1, \dots, N\}$. My goal is to divide $y_n$ in $K$ clusters and want my clusters to satisfy the following conditions:
Each ...
0
votes
0answers
16 views
Markov Chain vs Markov Jump Chain
I am unable to differentiate between "Markov Jump Chain Process" and "Markov Jump Process".
The mathematical notation is not sending me clear idea about this Jump Chain process.
...
0
votes
0answers
8 views
How to deal with interval-observed data in survival analysis?
Recently I've been doing some research with multi-state Markov survival models. Namely I have the data:
...
0
votes
1answer
23 views
stationary distribution of a markov chain
I am trying to find all the stationary distributions of the $3\times 3$ transition matrix below. because of the third state, the chain isn't irreducible however that isn't a sufficient condition for ...
0
votes
0answers
8 views
Is standard deviation of residuals of Markov switching GARCH model regime specific?
The MSGARCH package in R implements Markov switching GARCH model specified in the paper "A New Approach to Markov-Switching GARCH Models" (2004) by Haas et al.
By the code ...
0
votes
0answers
22 views
Understanding how to calculate removal effects in a markov chain
I am currently trying to model a Marketing Multi-Channel Attribution. All the articles and the packages I have come across use a special "start" state and the removal effect is calculated ...
0
votes
0answers
22 views
Markov property in time series
I work on some time series which are 1Hz. I have three questions about them:
How can I test Markov property on time series?
If Markov property doesn't hold, Is it possible to downsample time series ...
1
vote
0answers
20 views
does it make sense to define average reward in finite horizon
I am new to reinforcement learning but there is a situation I am considering using average reward instead of sum reward as objective for a finite horizon application problem.
Specifically, there are ...
1
vote
0answers
11 views
Covariate dependent Markov models? Plot state transition probability along gradient of covariate values
fist post here, came from Stack overflow as it was suggested to me this is more appropriate for the kind of question.
So, data consists of 4 variable, id, x1 and x2, continuous variables which are ...
2
votes
1answer
52 views
Help with Power Curve MCMC
I'm trying to analyze some $D$ non-logistic cumulative data in a time series, bounded below by 0 and unbounded above.
Splitting data into $W$ time windows of $d$ days, I know each window can be ...
0
votes
0answers
14 views
Chapman kolmogrovs equation diffrential vs integrated
I wasstudying about the markov chains and jumps properties and ways to solve and form equation about them when I came across the chapman kolmogrovs equation. So i just wanted to ask when should i use ...
0
votes
0answers
29 views
Gaussianity of Posterior required or Not with Fisher Information ? Validation by MCMC?
I am working on Fisher formalism and MCMC method.
First question) It seems that Fisher formalism assumes that posterior is always Gaussian : is it true ?
So if I find with MCMC a gaussian posterior, I ...
1
vote
0answers
15 views
Working out expected steps of absorbent Markov Chain with more than one sink
So I've seen answers regarding the relative probabilities of winning given that two people are tossing coins until one wins, with one person winning with HHT and the other with HTT. (2/3 HHT wins, 1/3 ...
1
vote
0answers
19 views
How to maximize the steady state transition probability for a state in a Markov chain by altering that state's outgoing transition probabilities?
Let's say we have a transition matrix of
which can be solved to come up with steady state transition probabilities of
Alpha: 34.9%
Beta: 24.0%
Gamma: 16.9%
Delta: 24.2%
Now, imagine Alpha, Beta, ...
0
votes
0answers
4 views
Error-Estimate using data with autocorrelation
I have generated some (high-dimensional) Datapoints $x_1,..., x_n$ using a Markov process that models some known probability-distribution $P(x)$. Now in order to estimate some expectation value $\...
0
votes
1answer
11 views
Markov Processes - question about an inference equation
in the following link, time 19:49:
https://www.youtube.com/watch?v=yOWBb0mqENw&list=PLdAoL1zKcqTXFJniO3Tqqn6xMBBL07EDc&index=2
there is the following equation:
$P(S_{t+k}|S_t)$ = $\sum_{s_{t+1}...
0
votes
0answers
17 views
Simulate discrete state space CTMC from generator matrix
Consider a generator matrix $Q\in\mathbb{R}^{h\times h}$ for a discrete state space $\{1,...,h\}$. I want to determine the probability of a single transition of a stochastic process $X(t)$ with $X(0)=...
0
votes
0answers
16 views
Creating a transition matrix based on a Markov chain in R
I have four distributions that represent incomes in R. I categorise them by what income group they fall under such as under half the mean, between half the mean and 3/4th of the mean and so on until ...
0
votes
1answer
28 views
Running several MCMC chains after convergence?
I am running a MCMC Gibbs sampler for a computationally expensive model. It takes ~12 hours to obtain 1000 iterations of this MCMC sampler. I have tested the sampler, and I found that the chain seems ...
0
votes
0answers
15 views
Volatility based Markow Switching GARCH model
I am trying to model returns using ARMA-GARCH process and noticed that returns series behave differently under the periods of high volatility when compared to periods with low volatility.
Therefore, I ...
0
votes
1answer
23 views
Understanding Bayes Rule Application in POMD Belief State Update
I am trying to wrap my head around Partially Observed Markov Decision Process (POMDP). However, I am unable to understand the application of the Bayes Rule in following equation (Step Nr. 2):
Can ...
1
vote
1answer
26 views
What does it mean if X ā Y ā Z is a markov chain, it mplies that Z ā Y ā X. Some-times written X ā Y ā Z
In the book Elements of information theory (Cover, Thomas) 2nd ed. Page 34
On Markov Chain it says:
...
1
vote
0answers
20 views
Training two Hidden markov models vs two state Hidden Markov models
I have a scenario where I have log of events followed by some kind of special event (e.g Failure etc). I have two kind of sequences (events, that are observations, can be common in both sequence), ...
