Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
0
votes
0answers
6 views
Data augmentation for animal measurements
I am looking for methods to do data augmentation for an animal study. What I want to do is generate a large number of data points by using a min and max measurements of the animal.
So I have 2 ...
0
votes
0answers
2 views
MLE with higher order markov chain model
I am trying to estimate my Markov chain model. I understand that in a standard model, a higher order will lead to overfitting and higher likelihood. Hence, one can use AIC/BIC to find the order of the ...
0
votes
0answers
13 views
How to get just date when the time comes like this? [on hold]
I want to get just the dates because I'm modeling my problem as a markov chain. I've already created a matrix with all the dates (without time) using a seq and I'll compare it later. How can I get ...
0
votes
1answer
15 views
Hidden States in Hidden Markov Model
I am using Hidden Markov Models, having observations as continuous variable and states as discrete variable. I can use the observations to train HMM model and generate n number of states(say 2 hidden ...
0
votes
0answers
40 views
Does the Markov property always hold for a state-space structure?
Markov Property: $p({\bf x}_t | {\bf x}_1, \ldots, {\bf x}_{t-1}) = p({\bf x}_t | {\bf x}_{t-1})$
Consider the following model for which the hidden states are ${\bf x}_t$ and the observations are ${\...
1
vote
2answers
50 views
Irreducible markov chain problem
Let $P$ be a stationary transition probability matrix of the markov chain $ \{X_n , n \ge 0\}, $ which is irreducible and every state has a period 2. Further suppose that the markov chain $\{Y_n , n \...
1
vote
0answers
21 views
How to increase the total number of iterations it takes to converge a MDP?
I was reading about Policy Iteration. What are the factors that influence the total number of iterations the algorithm takes to converge?
For a given MDP which converges in 3 iterations, what setting ...
0
votes
1answer
13 views
Is this Markov Chain calculation correct?
$S=\{1,2\}$
$\alpha = (1/2, 1/2)$
$P= \begin{bmatrix}
1/2&1/2\\
0&1\end{bmatrix} $
Find
$P(X_1=1 | X_0=1)$
Given solution:
$P(X_1=1 | X_0=1) = \frac{P(X_1=1 , X_0=1)}{P(...
0
votes
0answers
13 views
Programing a discrete time Markov chain simulation in R with individual-specific transition intensities
I need to simulate a discrete time Markov chain with individual-specific transition intensities (no differential equations) to model the prevalence of an infectious disease in an ageing population. I ...
0
votes
0answers
17 views
Markov chains and Equilibrium distribution
Two yachts, sailed by "Yacht 1" and "Yacht 2" respectively are sailing around a course. If the teams are even at the beginning of a lap then during that lap they embark on a duel and one team gains a ...
0
votes
0answers
10 views
Training Hidden Markov model with GMM, nan appears after some iterations
Problem
During the training process of my continuous observation sequence data using HMM with GMM mixtures, the cost function reduces gradually and it becomes NaN ...
0
votes
0answers
18 views
conversion prediction / ROC curve using markov chains for channel attribution
I am currently working on a project on multi-channel attribution, using the channel attribution package from Altomare & Loris (2018), which uses markov chains for attribution. A walk-through of ...
0
votes
0answers
13 views
Geometric distribution with multiple success state and markovian succes probability
Let $X_t$ be a random process in the space $E:=\{F, S_1, S_2, S_3\}$ for each $t$. We can see it like it is a game where we can win in three different ways or we can fail. We play until we fail for ...
0
votes
0answers
13 views
Interrogating the results of the Markov simulation - Help and feedback highly appreciated
0
I have built a Markov chain with which I can simulate the daily routine of people (activity patterns). Each simulation day is divided into 144-time steps and the person can carry out one of ...
0
votes
2answers
34 views
understanding time-homogeneous markov chain
Could anyone help me understand the definition on page 7 definition 2.25 here? I do not understand the notation $(P(a))(A)$ - what does this mean?
Also, is $P(a, A)$ a probability measure from the ...
2
votes
0answers
19 views
What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?
The expected average reward for a policy $\pi$ is:
$$ \rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$
where $r_t$ is the reward obtained at time $t$ following policy $\pi$.
...
0
votes
0answers
11 views
Markov Chain for Different Groups of Time Series
I would like to use (hidden) Markov Chain to predict $X[t+1]$ stock price. Historical data for top biggest 500 companies for last 10 years will be used for training.
The error would be sum of $...
2
votes
1answer
50 views
Metropolis-Hastings - interpreting the transition kernel: alpha*proposal
I thought I had great intuition and mathematical understanding of the Metropolis-Hastings algorithm, until closer inspection... as I started compiling my notes, I realized I do not understand the ...
1
vote
0answers
23 views
Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions
I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$.
Now my idea is to run ...
0
votes
0answers
9 views
Zero Sum problem with MDP formulation and the difference with minimax approach
Suppose that we have formulate a zero sum game with MDP and Ua(s) and Ub(s) are the utilities of A and B in the s state.
Suppose that all rewards and utilities are calculated from the A's point of ...
1
vote
1answer
32 views
How can I compute $P(X_1 = 1|X_0 = 1)$ from the given data?
I know how to mathematically calculate the probability of various Markov Properties.
But, how can I calculate Markov probability from data?
Suppose, I have a Markov Chain as follows:
$S=\{1, 2\}$
...
2
votes
0answers
29 views
Understanding Approximate Dynamic Programming
I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
1
vote
1answer
25 views
Markov Blanket of two nodes?
I'm trying to solve a question and it has asked for (I feel like I'm confused by everything, would apprecaite some help, thanks),
the Markov blanket of {c,d}. From what I've read so far, Markov ...
0
votes
1answer
25 views
MCMC changing multiple parameters
What is the standard way of drawing multiple parameters for a MCMC run? Say I have 9 parameters, what is the most efficient method to get all parameters to explore their distributions properly?
1
vote
1answer
39 views
What happens if the observations are connected in a hidden Markov model (HMM)?
Suppose that we have an HMM with hidden variables $X_t$ and observed variables $Y_t$. Why do we always assume $p(Y_t|X_t)$? What happens if we have $p(Y_t|X_t, Y_{t-1})$? Is it because that wouldn't ...
2
votes
1answer
50 views
Manual simulation of Markov Chain in R
Consider the Markov chain with state space S = {1, 2}, transition matrix
and initial distribution α = (1/2, 1/2).
Simulate 5 steps of the Markov chain (that is, simulate X0, X1, . . ...
0
votes
1answer
30 views
Hidden Markov Model probability of producing a sequence
Suppose that we have two models for a 2-state HMM and both have two output symbols: $A$ and $B$.
Model 1:
Transition probabilities: $a_{11}=0.6$, $a_{12}=0.4$, $a_{21}=0.0$, $𝑎_{22}=1.0$.
Output ...
1
vote
1answer
57 views
For a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?
I am new to Markov Chains and using this concept in statistics.
For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?
If yes, how can I prove that?
0
votes
1answer
42 views
limit and stationary distribution of a Markov chain
Consider a Markov chain on the non-negative integers with transition
probabilities $1/2$ if $y=x+1$ and $1/2$ if $y=0$. Find $\lim_{n \to \infty} P(X_{n}=0)$.
Is this limit the same as the ...
0
votes
0answers
11 views
Countably infinite state space for Continuous Time Markov Chains
I have been unsuccessfully browsing the internet for resources related to Continuous Time Markov Chains with a countably infinite state space, for example $\mathbb{N}$.
Is there any developed theory ...
4
votes
2answers
85 views
Why does Judea Pearl call his causal graphs Markovian?
In his texts on causality, Judea Pearl always refers to the simplest graphs he uses, i.e. the acyclic graphs with independent confounders, as Markovian. I don't see why these graphs contain anything ...
0
votes
1answer
27 views
Compute the Limiting Distribution
Consider the transition matrix
$ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $
for general $2$-state Markov Chain $(0 \le p, q\le 1)$.
Find the limiting distribution ...
0
votes
0answers
12 views
Model selection with this model of a large number of components
I have a discrete time Markov Chain $\{X_n: n \in \mathbb{N}_0\}$ with unknown transition matrix $P \in \mathbb{R}^{M \times M}$ on the state space $\mathcal{S}_X = \{1,2, \dots, M\}$, with $M \geq 2$....
1
vote
1answer
23 views
How can I compute expected return time of a state in a Markov Chain?
I was watching a YouTube video regarding the calculation of expected return time of a Markov Chain.
https://www.youtube.com/watch?v=X_Ll0-Ytu7U&vl=en
I haven't understood the calculation of $m_{...
0
votes
0answers
14 views
Understanding the meaning of transition rates in a CTMC
I am reading the queueing theory volume 1 by Kleinrock. In the chapter on Continuous Time Markov Chain(CTMC), the author defines the infinitesimal generator, $Q(t)$ as having the following elements:
$...
1
vote
0answers
28 views
Markov property of manifest variables in longitudinal ESM
I am working on a longitudinal ESM model were the indicators are (highly) autocorrelated. This means that the classic cross-lagged models of panel data analysis cannot be used directly. I have ...
1
vote
0answers
19 views
Reversible markov chain
i have a question from a mid session exam. I was marked incorrect and I dont understand why. Question: Given a reversible markov chain P, with measure $\pi$. Show that $P^2$ is reversible.
Since P ...
0
votes
0answers
20 views
Markov Chain - Identifying the customers from model attribution results
I'm using Markov Chains to model a customers journey. My purpose is to attribute weights to channels, e.g. TV, FB, Billboards etc., for their relative importance.
For example, one arbitrary customer ...
0
votes
1answer
30 views
How to Show that a Distribution is a Stationary Distribution for Metropolis-Hastings? [closed]
For an Ising Model with a (2L+ 1) by (2L+ 1) square grid
of magnetic particles, show that $$\pi(\xi)=\frac{1}{Z_\beta}e^{\beta\sum_{x,y=x}{\xi_x\xi_y}}$$ Is indeed a stationary distribution for the ...
0
votes
0answers
38 views
How to use the Expectation Maximization (EM) algorithm for Part of Speech (POS) tagging?
I want to know how can we use the EM algorithm for Part of Speech (POS) tagging.
The data is a set of sentences Xs and their POS tags Ys i.e. a sentence X is a sequence of words $(X_1,X_2,\ldots, ...
0
votes
1answer
31 views
Kernel for a markov process?
Could anyone just explain to me what does it mean by mathematically, $P_n(x, dy)$ is the law of $X_n$ here in the page $46$.
https://statweb.stanford.edu/~cgates/PERSI/papers/iterate.pdf
Thanks for ...
1
vote
1answer
70 views
MCMC - target distribution, proposal distribution and likelihood function?
i have just started out in MCMC and I am not sure if I fully understand the concepts of MCMC with respect to the above terms. Let me try to explain that in my own words and include some thoughts / ...
5
votes
1answer
45 views
What does the distribution of samples from an MCMC method converge to without repeated samples?
Suppose I have an absolutely continuous distribution with density $f(x)$ and I use an mcmc sampler which has accept/reject step to sample from this distribution. In the final samples, there are some ...
0
votes
0answers
234 views
Markov chain: how to estimate the transition matrix? I don't have the underlying observations, just the sum by state and time
I have a matrix for data that (supposedly) follows a Markov process with an absorbing state; I have 3 possible states and 50 periods (discrete states, discrete time). Element [t,s] of the matrix tells ...
0
votes
1answer
29 views
What's the proper name for these chain structured PGMs?
I'm trying to find previous work that has dealt with this type of PGMs, but don't know what to call them:
a) "recurrent HMM"? $y_i$ are scalars and $x_i$ are discrete
b) "triangle HMM"? again, $y_i$ ...
3
votes
1answer
201 views
How does the Metropolis Algorithm “get off the ground”?
I'm thoroughly confused by the Metropolis Algorithm as defined in Casella and Berger's Statistical Inference. Namely, here's the definition (p.254):
Let $Y \sim f_Y(y)$ and $V \sim f_V(v)$, where $...
0
votes
1answer
26 views
Long run proportion of transitions in a Markov chain
Let $S$ be a set of states for a Markov chain and let $S^C$ be the remaining states. Explain the identity $$\sum_{i\in S}\sum_{j\in S^C}\pi_iP_{ij}=\sum_{i\in S^C}\sum_{j\in S}\pi_iP_{ij}$$
I know ...
1
vote
1answer
116 views
MCMC autocorrelation
I have a MCMC simulation that tries to fit a line to a linear set of data. The auto-correlation is very high for the slope parameter (~0.9), and low (~0.05) for the bias
What does a high auto-...
6
votes
1answer
88 views
How can we verify the intuition that in the RW-Metropolis-Hastings algorithm with Gaussian proposal too small and too large variances are bad choices
Let $d\in\mathbb N$ and consider the Random Walk Metropolis-Hastings algorithm with a Gaussian proposal kernel $Q$ such that $Q(x,\;\cdot\;)=\mathcal N_d(x,\sigma^2_dI_d)$ for all $x\in\mathbb R^d$.
...
0
votes
1answer
62 views
Relation between Uniform distribution, Metropolis Algorithm, and Symmetric Proposal Distribution
I am having some confusion over the Metropolis algorithm. Let $g(x|y)$ be our proposal distribution for the algorithm. For the Metropolis, $g$ must be symmetric (from Wikipedia). In the discrete case, ...
