Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
0
votes
0answers
39 views

How to use the Expectation Maximization (EM) algorithm for Part of Speech (POS) tagging?

I want to know how can we use the EM algorithm for Part of Speech (POS) tagging. The data is a set of sentences Xs and their POS tags Ys i.e. a sentence X is a sequence of words $(X_1,X_2,\ldots, ...
0
votes
1answer
32 views

Kernel for a markov process?

Could anyone just explain to me what does it mean by mathematically, $P_n(x, dy)$ is the law of $X_n$ here in the page $46$. https://statweb.stanford.edu/~cgates/PERSI/papers/iterate.pdf Thanks for ...
1
vote
1answer
81 views

MCMC - target distribution, proposal distribution and likelihood function?

i have just started out in MCMC and I am not sure if I fully understand the concepts of MCMC with respect to the above terms. Let me try to explain that in my own words and include some thoughts / ...
5
votes
1answer
45 views

What does the distribution of samples from an MCMC method converge to without repeated samples?

Suppose I have an absolutely continuous distribution with density $f(x)$ and I use an mcmc sampler which has accept/reject step to sample from this distribution. In the final samples, there are some ...
0
votes
0answers
339 views

Markov chain: how to estimate the transition matrix? I don't have the underlying observations, just the sum by state and time

I have a matrix for data that (supposedly) follows a Markov process with an absorbing state; I have 3 possible states and 50 periods (discrete states, discrete time). Element [t,s] of the matrix tells ...
0
votes
1answer
30 views

What's the proper name for these chain structured PGMs?

I'm trying to find previous work that has dealt with this type of PGMs, but don't know what to call them: a) "recurrent HMM"? $y_i$ are scalars and $x_i$ are discrete b) "triangle HMM"? again, $y_i$ ...
3
votes
1answer
202 views

How does the Metropolis Algorithm “get off the ground”?

I'm thoroughly confused by the Metropolis Algorithm as defined in Casella and Berger's Statistical Inference. Namely, here's the definition (p.254): Let $Y \sim f_Y(y)$ and $V \sim f_V(v)$, where $...
0
votes
1answer
28 views

Long run proportion of transitions in a Markov chain

Let $S$ be a set of states for a Markov chain and let $S^C$ be the remaining states. Explain the identity $$\sum_{i\in S}\sum_{j\in S^C}\pi_iP_{ij}=\sum_{i\in S^C}\sum_{j\in S}\pi_iP_{ij}$$ I know ...
1
vote
1answer
152 views

MCMC autocorrelation

I have a MCMC simulation that tries to fit a line to a linear set of data. The auto-correlation is very high for the slope parameter (~0.9), and low (~0.05) for the bias What does a high auto-...
6
votes
1answer
91 views

How can we verify the intuition that in the RW-Metropolis-Hastings algorithm with Gaussian proposal too small and too large variances are bad choices

Let $d\in\mathbb N$ and consider the Random Walk Metropolis-Hastings algorithm with a Gaussian proposal kernel $Q$ such that $Q(x,\;\cdot\;)=\mathcal N_d(x,\sigma^2_dI_d)$ for all $x\in\mathbb R^d$. ...
0
votes
1answer
65 views

Relation between Uniform distribution, Metropolis Algorithm, and Symmetric Proposal Distribution

I am having some confusion over the Metropolis algorithm. Let $g(x|y)$ be our proposal distribution for the algorithm. For the Metropolis, $g$ must be symmetric (from Wikipedia). In the discrete case, ...
0
votes
1answer
18 views

Markov Chain Question/Notation Confusion

Show that if $(X_n)_{n \geq 0}$ is a discrete-time Markov chain with transition matrix $P$ and $Y_n = X_{kn}$, then $(Y_n)_{n \geq 0}$ is a Markov chain with transition matrix $P^k$. I am a little ...
3
votes
1answer
100 views

How can we numerically compute the autocorrelation of a sample from a Markov chain generated by the Metropolis-Hastings algorithm?

Let $(X_n)_{n\in\mathbb N_0}$ denote a $\mathbb R^d$-valued Markov chain generated by the Metropolis-Hastings algorithm. Suppose I've run the algorithm on a computer and obtained a sample $x_0,\ldots,...
0
votes
0answers
13 views

Calculate Transition Probabilities Interest rate data

I came across a paper by Rodda (2004), who simulates interest rates with a Markov sequence. To simulate changes in the interest rates, they used the historical transition probabilities. Their ...
2
votes
0answers
63 views

Numerical examples proving and disproving the optimal scaling heuristic by Roberts et al

Let $d\in\mathbb N$ with $d>1$ $\ell>0$ $\sigma_d^2:=\frac{\ell^2}{d-1}$ $f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$ $Q_d$ be a Markov kernel on $(\mathbb R^...
0
votes
0answers
62 views

Transition probability in Markov chain using a decision tree model

I wish to find a way to calculate the transition probabilities in my Markov chain model. Let's say a customer has three products [A B C] and in this example I wish to know the transition probability ...
5
votes
1answer
100 views

Condition on the covariance matrix of a gaussian process needed to have the Markov property

Let suppose to have a realization $\mathbf{X}=(\mathbf{X}_1,\dots, \mathbf{X}_n)$, where $\mathbf{X}_i \in \mathcal{R}^d$, from a $d-$variate Gaussian process. Let also suppose that $E(\mathbf{X}_i)= ...
0
votes
0answers
12 views

How to calculate nth step in the binary sequence?

I have a dataset with 499 observations for a single binary variable. The objective is to predict the next observation in the series. Here is the dataset: 1 2 1 2 1 1 2 1 2 1 1 1 1 1 1 1 1 2 2 1 1 1 ...
5
votes
0answers
92 views

How does the celebrated result about the diffusion limit of the Random Walk Metroplis-Hastings algorithm help us to find the optimal scaling

Let $d\in\mathbb N$ with $d>1$ $\ell>0$ $\sigma_d^2:=\frac{\ell^2}{d-1}$ $f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$ $Q_d$ be a Markov kernel on $(\mathbb R^...
0
votes
0answers
24 views

Sequence Prediction with noise / gap using markov models

I'm trying to understand if Markov models can account for a "noise event" when predicting the next item in a sequence. For instance, if i have very frequently occurring (noise) event "F", can a ...
0
votes
0answers
49 views

What is the difference between homogeneous Markov chains and unhomogeneous Markov chains?

I learned that a Markov chain is a graph that describes how the state changes over time, and a homogeneous Markov chain is such a graph that its system dynamic doesn't change. Here the system dynamic ...
1
vote
0answers
82 views

Markov Models for time series prediction

I am student conducting an experiment with different models for time series prediction. In my experiment, I am going to use ARIMA, a Recurrent Neural Network, a Long-Short Term Memory network, and a ...
1
vote
0answers
29 views

Finding Probability of a digit given a sequences?

I have a n sequences of numbers ranges from 1 to 4, say sequence s1 = [1,3, 1, 4] and s2 = [2,1,3,4] up to s(n). My question is how can i find probability of a number coming right after a sequence, ...
1
vote
1answer
41 views

Discrete-time Markov Chain; $n$-step transitions

Let $\{X_{n}\}_{n\geq0}$ be a discrete-time Markow chain on the state space $S=\{1,2,3\}$ with transition matrix \begin{pmatrix} 1/3 & 1/3 & 1/3 \\ 0 & 2/3 & 1/3 \\ 2/3 & 1/...
2
votes
0answers
25 views

Trouble understanding derivation of probability for continuous time markov chain

I'm working on exercise 6.10 from "Introduction to probability models" by Sheldon M. Ross. There's an expression for the probability $P_{00}(t)$ that I don't understand. Here's the relevant ...
0
votes
0answers
30 views

Entropy/Measure of Knowledge for probability intervals

I have a system which outputs probability interval tuples over the decision set $\{d_1, d_2, d_3\}$, such that a tuple $([\min_1, \max_1], [\min_2, \max_2], [\min_3, \max_3])$ indicates that $\min_i$ ...
2
votes
2answers
161 views

Visualization of the number of transitions between states [closed]

I am currently developing a Markov model for ordinal data. In order to proceed with the modeling, I would like to check the distribution of the number of transitions per individual in my data set. ...
1
vote
0answers
46 views

How to use trial and error algorithm to predict the next number in a sequence?

I have a time series data. I want to use trial and error algorithms to predict the next number in a variation_sequence. I mean about Trial and error algorithm is using an online learning and where I ...
0
votes
0answers
23 views

Optimal strategies in a memoryless MDP

Consider the following theorem: Let $M$ be a finite MDP with state space $S$ and $B \subset S$. There exist a memoryless deterministic scheduler $\Theta^{min}, \Theta^{max}$ such that for any $s \...
0
votes
0answers
22 views

Bootstrap resulted in something that looks like a mixed distribution. What do?

TLDR Bootstrapping resulted in a crazy scatterplot. Totally clueless here. I have a dataset (not going to say much about it, it is a bit confidential), running a Markov chain-based algorithm on the ...
1
vote
1answer
54 views

Number of parameters in an Mth Order Markov Chain

For an $M^{th}$ order Markov chain, $P\left(X_n|X_{n-M}...X_{n-1}\right)$, what's the number of parameters required to know the conditionals? We have discrete variables each with $K$ states. I think ...
1
vote
1answer
79 views

Irreducible (communicating) classes [closed]

The Markov chain $(Xn; n\geq)$ has state-space $S = (0, 1, 2, . . .)$, with $p_{i,0} = \frac{1}{4}$ and $p_{i,i+1} = \frac{3}{4}$ $\forall i \geq 0$, so that the transition matrix is P =$\...
0
votes
1answer
61 views

Biased coins and Markov processes

Good day, I am attempting an optional exercise and I am finding it hard to interpret the problem in terms of matrices and vectors. Coin 1 has probability 0.4 of coming up heads, and coin 2 has ...
0
votes
1answer
34 views

What does environment dynamic means in Reinforcement learning

From the book Reinforcement learning: an introduction, I have two questions: 1) there is the following sentence: "If the environment's dynamics are completely known, then finding the optimal policy ...
0
votes
1answer
111 views

Probabilistic user behavior markov models on web

I am considering the following probabilistic Markov model of actions of a user on the results page of a search engine. The user examines the first result, with a probability $A$ he is satisfied with ...
1
vote
0answers
71 views

States of Markov chain and stationary distribution

Let $X$ be a Markov chain with a state space $S={\{0,1,2,... \}}$ and a transition matrix $P$ with given $p_{i,0}=\frac{i}{i+1}$ and $p_{i,i+1}=\frac{1}{i+1}$, for $i=0,1,2,...$. Find out which states ...
3
votes
1answer
134 views

Proving that given Markov chain is homogeneous. Find state space and transition matrix

Let $X_i$ be the results of a consecutive throws of a die. Let $Z_n=3(X_1^2+\cdots+X_n^2) \bmod 5$. Show that the sequence ${\{Z_n \mid n\geq1\}}$ is a homogeneous Markov Chain. Find a state space and ...
1
vote
0answers
85 views

Expected number of steps in Gambler's ruin game with two players

Let's say we have two players A and B. Player A has 3 coins and player B has 5 coins. If player wins the other player gives one coin. During game second player probability of loosing is $2/3$, while ...
1
vote
1answer
111 views

Why is variational Bayesian mixture model an alternative to MCMC? What are the similarities?

Why do people say that a variational Bayesian mixture model could be an alternative to MCMC for clustering? For example see the details here: https://en.wikipedia.org/wiki/Variational_Bayesian_method. ...
2
votes
0answers
29 views

Importance of the right-continuity of filtration in definition of strong Markov Property

Taking the definition from wikipedia, With $X = (X_t : t \geq 0) $ as a stochastic process on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$ with natural filtration $\{ \mathcal{F}(t) \}_{t \...
0
votes
1answer
36 views

Can a state have zero periodicity? [closed]

I am getting my concepts cleared in Stochastic process. I understand the concept of periodicity. Just to make it clear, suppose there is a finite Markov chain with states $1,2,3$. Let their ...
1
vote
1answer
66 views

Why is the probability of a random walk reaching 1 (in n steps) squared greater than the probability of it reaching 2 (in n steps)?

Let $S_n$ be a simple random walk. i.e. $$ S_n = \sum_{t=1}^n X_t, $$ where ${X_t}$ are i.i.d random variables with $$ X_t = \begin{cases} +1, & \textrm{w/ probability } p \\ -1, & \...
1
vote
0answers
27 views

Convergence in total distribution distance in the Random Walk Metropolis-Hastings algorithm

I'm searching for a proof of the convergence in total distribution distance of the transition probabilities of a Markov chain generated by the Random Walk Metropolis-Hastings algorithm to its ...
0
votes
1answer
53 views

Generating very few samples from a probability distribution using MCMC?

Since MCMC converges to target only after taking very large number of steps, what if I want to have just say 10 samples from target distribution? Do I still have to generate lots of samples, and then ...
2
votes
0answers
81 views

Estimation of transition probability matrix (TPM) for a discrete time, continuous state markov chain from uniformly-spaced samples

I have uniformly spaced samples from a three-component (i.e. three nodes) Markov chain: $s^{(0)}=\begin{bmatrix}0.99\\ 0.01\\ 0.00\end{bmatrix}$, $s^{(1)}=\begin{bmatrix}0.98\\ 0.01\\ 0.01\end{...
6
votes
1answer
164 views

Optimal scaling of the Random Walk Metroplis-Hastings algorithm and the speed measure of the limiting diffusion

Let $d\in\mathbb N$ with $d>1$ $\ell>0$ $\sigma_d^2:=\frac{\ell^2}{d-1}$ $f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$ $Q_d$ be a Markov kernel on $(\mathbb R^...
0
votes
0answers
11 views

Residence times of the telegraph process ?

The telegraph process is a two state stochastic process defined by the master equation $$ \dot{\pi}_0(t) = \tau^{-1} \pi_1(t) - \sigma^{-1} \pi_0(t) $$ $$ \dot{\pi}_1(t) = \sigma^{-1} \pi_0(t) - \...
3
votes
0answers
62 views

Estimating a MS-ARMA(p,q)-GARCH(r,s) parameters via MCMC

I am currently working on a MS-ARMA-GARCH model proposed by Dhiman das on this paper, and trying to fit it on simulated data. So far I understand the model and its construction, but I'm having a hard ...
1
vote
0answers
32 views

How to create the initial ensemble samples for EnKF

As we know, for the ensemble Kalman filter (EnKF), we need to create a set of samples in the beginning and then to run the predict and analysis step. But for now I have a question of how to create the ...
0
votes
0answers
87 views

Formulating a Transition matrix for Markov Process

I am dealing with a medical process which is as follows. There are 10000 Veterans who are enrolled in this study. All 10000 have medical condition called onychocryptosis which is a fancy term for ...