# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,109 questions
Filter by
Sorted by
Tagged with
7 views

### Strategic Multi Armed Bandit

As a part of my project, I have been tasked with formulating a multi-armed bandit problem with strategic arms. What I have found out is a Gittin's index approach to the problem provides a solution ...
20 views

### How to test that a discrete time series has markovian properties? How to use the Wilcoxon test to do so?

Let us assume I have a time series made of the following observations: ts = c(163,18,53,189, 243, 101, 150, 39, 60,96,36,76,71,67,56,3,72,96,15,19) How can I use the Wilcoxon test to determine if it ...
563 views

### Equilibrium distribution of Markov chain

The transition matrix is P =\begin{bmatrix} \frac12 & \frac12 & 0 & 0 \\ \frac12 & \frac12 & 0 & 0 \\ 0 & 0 & \frac13 & \frac23 \\ 0 & 0 & \frac13 & \...
63 views

### Long term expected win rate

I have the following game, you have some initial probability $p_i$ to win, every time you try and lose the probability increases by a constant $p_a$ and is reset to $p_i$ when you win. For example if ...
11 views

### Gibbs sampling Bayesian conditional distribution for mean of a Normal distribution

first post here in CV. I'm currently working on a textbook exercise on Gibbs Sampling and got stuck on naming the distribution for one of the conditional distributions. Question Consider a normal ...
25 views

### Gambling game: If he wins he quit if he loses doubles the bet [closed]

A player is gambling based on the following strategy. The player bets $\$1$at first round. If he wins, he ends the game. If he loses, he doubled the bets to$\$2$ at the second round. If he wins, he ...
122 views

### Markov chain - ergodic theorem

I need help with transition matrix of the following problem On each evening, the owner of a car washes it with probability 0.6. Independently of that event, every night a dirty rain pours on the car ...
27 views

### Convergence of policy iteration with discount $\gamma=1$

This question is related to my question (and my comment of RobPratt's answer) at https://math.stackexchange.com/questions/3860303/markov-decision-process-with-target-states-and-shortest-path-as-only-...
14 views

### PDF of a two state Markov chain with binned observation

I am trying to come up with a PDF to describe experimental data, which I can describe well with a simple Monte Carlo simulation. I have a two state Markov chain with equal transition probabilities ...
31 views

### expected number of dependent trials until n successes

In case of bernoulli experiments, I know that the expected value for the number of trials needed to have $n$ successes is n/p. (where p is probability of success for each trial). Now my question is, ...
6k views

### A fair die is rolled 1,000 times. What is the probability of rolling the same number 5 times in a row?

A fair die is rolled 1,000 times. What is the probability of rolling the same number 5 times in a row? How do you solve this type of question for variable number of throws and number of repeats?
25 views

### Predicting Graph Edge Connections

I have a set of nodes in 3d physical space. Some of those nodes are connected to one another by a graph edge, while others are not. Just because two nodes are physically close doesn't necessarily mean ...
54 views

### How to determine if a -long- time series made of discrete observations is markovian?

Let us assume I have a time series made of the following observations: ts = c(163,18,53,189, 243, 101, 150, 39, 60,96,36,76,71,67,56,3,72,96,15,19) How can I determine if it respects the Markov ...
35 views

### Calculation of transition probabilities of Markov Chain

I have just started learning markov chains and I need help on the following question: Alice and Bob vote in each parliamentary election. If, in a certain election, Alice and Bob vote for the same ...
3 views

### Element by element t-test between two dependent Markov transition matrices

I want to conduct t-tests between each of the probabilities of an unconditional matrix and a conditional matrix generated from discrete-time Markov chains. The unconditional probabilities are derived ...
35 views

### Is a policy $\pi(s)$ on Markov decision process a random variable?

Citing Wikipedia: The goal in a Markov decision process is to find a good "policy" for the decision maker: a function $\pi$ that specifies the action $\pi(s)$ that the decision maker will ...
7 views

### Statistically compare transition probabilities from two transition matrices

I have a dataset that is stratified by gender. I have fit a first-order Markov chain to both strata, giving transition matrices $M_1$ and $M_2$. Now I want to statistically test, for given $i, j$, ...
34 views

### How to solve a Markov Decision Problem with State Transition Matrix and Reward Matrix

I'm stuck in solving a simple dynamic probabilistic model. I have Three states {Sunny, Cloudy, Rainy}. I have the ...
19 views

### state transition classification on terminal state

see https://datascience.stackexchange.com/questions/82169/state-transition-classification-on-terminal-state 0 I have data on a unit 𝑖 which enters an entry state 𝑆0. This unit has some covariates 𝑥�...
48 views

### Estimating conditional probabilities in the context of finite state discrete time Markov chains of order p

Consider a Markov chain with evenly observed discrete times, where the order of the Markov chain is 1. Then we may estimate the transition probability of the Markov chain using sample proportions. ...
17 views

### Does it make sense to make a conditional Markovianity assumption?

Please do excuse if the title is somewhat vague. As we know, a discrete-time Markov chain is a sequence of random variables $S_1,S_2,\cdots$ with the Markov property - i.e. \begin{equation} P[S_{t+1}=...
31 views

### How can I test if a time-series data satisfies Markov's property and it is a martingale?

My question is about investigating some properties of time-series. How can I test if my time-series data satisfies Markov's property? How can I test if my time-series data is a martingale? I wonder ...
18 views

### Yule (preferential attachment process vs. birth&death process

What is the distinction between the Yule (preferential attachment) process and the birth&death process? Are they the same thing, called differently in different contexts, or is the birth&death ...
13 views

### How to “de-mean” a series so it can be used in MSGARCH R package?

0 I would like to use MSGARCH package to fit a Markov switching GARCH model. I know that there is a useful MSGARCH package in R but it does not allow to add a conditional mean equation like in rugarch ...
40 views

### How to calculate Markov Chain parameters knowing AR(1) coefficient $\rho$?

Given that a discrete-time two-states (low and high) Markov chain $X_t$ can take two values: \begin{pmatrix} X_l\\ X_h \end{pmatrix} with its transition probability matrix $P^{T}$: \begin{pmatrix} p_1 ...
66 views

### Carré du champ operator is a quadratic variation

Let $X_t$ be a real valued Markov process (starting at $x$) with generator $L$. Let $\Gamma(f)$ denote Carré du champ operator i.e. $L(f^2) - 2f \cdot L (f)$. As far as I know under suitable ...
26 views

### Markov Switching Model with Markov trend

I have a time series with nonstationary data. Looking at the plot of the data it is clear that there are structural breaks. As data follow the stochastic process, I want to build Markov Switching ...
18 views

### How can I cluster sequential data?

Suppose that I have a sequence of vectors $y_n \in \mathbb{R}^m$ for $n \in \{1, \dots, N\}$. My goal is to divide $y_n$ in $K$ clusters and want my clusters to satisfy the following conditions: Each ...
16 views

### Markov Chain vs Markov Jump Chain

I am unable to differentiate between "Markov Jump Chain Process" and "Markov Jump Process". The mathematical notation is not sending me clear idea about this Jump Chain process. ...
8 views

### How to deal with interval-observed data in survival analysis?

Recently I've been doing some research with multi-state Markov survival models. Namely I have the data: ...
23 views

### stationary distribution of a markov chain

I am trying to find all the stationary distributions of the $3\times 3$ transition matrix below. because of the third state, the chain isn't irreducible however that isn't a sufficient condition for ...
8 views

### Is standard deviation of residuals of Markov switching GARCH model regime specific?

The MSGARCH package in R implements Markov switching GARCH model specified in the paper "A New Approach to Markov-Switching GARCH Models" (2004) by Haas et al. By the code ...
22 views

### Understanding how to calculate removal effects in a markov chain

I am currently trying to model a Marketing Multi-Channel Attribution. All the articles and the packages I have come across use a special "start" state and the removal effect is calculated ...
22 views

### Markov property in time series

I work on some time series which are 1Hz. I have three questions about them: How can I test Markov property on time series? If Markov property doesn't hold, Is it possible to downsample time series ...
20 views

### does it make sense to define average reward in finite horizon

I am new to reinforcement learning but there is a situation I am considering using average reward instead of sum reward as objective for a finite horizon application problem. Specifically, there are ...
11 views

### Covariate dependent Markov models? Plot state transition probability along gradient of covariate values

fist post here, came from Stack overflow as it was suggested to me this is more appropriate for the kind of question. So, data consists of 4 variable, id, x1 and x2, continuous variables which are ...
52 views

### Help with Power Curve MCMC

I'm trying to analyze some $D$ non-logistic cumulative data in a time series, bounded below by 0 and unbounded above. Splitting data into $W$ time windows of $d$ days, I know each window can be ...
14 views

### Chapman kolmogrovs equation diffrential vs integrated

I wasstudying about the markov chains and jumps properties and ways to solve and form equation about them when I came across the chapman kolmogrovs equation. So i just wanted to ask when should i use ...
29 views

### Gaussianity of Posterior required or Not with Fisher Information ? Validation by MCMC?

I am working on Fisher formalism and MCMC method. First question) It seems that Fisher formalism assumes that posterior is always Gaussian : is it true ? So if I find with MCMC a gaussian posterior, I ...
15 views

### Working out expected steps of absorbent Markov Chain with more than one sink

So I've seen answers regarding the relative probabilities of winning given that two people are tossing coins until one wins, with one person winning with HHT and the other with HTT. (2/3 HHT wins, 1/3 ...
19 views

### How to maximize the steady state transition probability for a state in a Markov chain by altering that state's outgoing transition probabilities?

Let's say we have a transition matrix of which can be solved to come up with steady state transition probabilities of Alpha: 34.9% Beta: 24.0% Gamma: 16.9% Delta: 24.2% Now, imagine Alpha, Beta, ...
4 views

17 views

### Simulate discrete state space CTMC from generator matrix

Consider a generator matrix $Q\in\mathbb{R}^{h\times h}$ for a discrete state space $\{1,...,h\}$. I want to determine the probability of a single transition of a stochastic process $X(t)$ with \$X(0)=...
16 views

### Creating a transition matrix based on a Markov chain in R

I have four distributions that represent incomes in R. I categorise them by what income group they fall under such as under half the mean, between half the mean and 3/4th of the mean and so on until ...
28 views

### Running several MCMC chains after convergence?

I am running a MCMC Gibbs sampler for a computationally expensive model. It takes ~12 hours to obtain 1000 iterations of this MCMC sampler. I have tested the sampler, and I found that the chain seems ...
15 views

### Volatility based Markow Switching GARCH model

I am trying to model returns using ARMA-GARCH process and noticed that returns series behave differently under the periods of high volatility when compared to periods with low volatility. Therefore, I ...
23 views

### Understanding Bayes Rule Application in POMD Belief State Update

I am trying to wrap my head around Partially Observed Markov Decision Process (POMDP). However, I am unable to understand the application of the Bayes Rule in following equation (Step Nr. 2): Can ...