Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
0
votes
0answers
11 views
Interrogating the results of the Markov simulation - Help and feedback highly appreciated
0
I have built a Markov chain with which I can simulate the daily routine of people (activity patterns). Each simulation day is divided into 144-time steps and the person can carry out one of ...
0
votes
2answers
29 views
understanding time-homogeneous markov chain
Could anyone help me understand the definition on page 7 definition 2.25 here? I do not understand the notation $(P(a))(A)$ - what does this mean?
Also, is $P(a, A)$ a probability measure from the ...
2
votes
0answers
19 views
What is the relation between expected average reward and single step mean reward for a non-stationary MDP policy?
The expected average reward for a policy $\pi$ is:
$$ \rho_\pi = \lim_{T \rightarrow \infty } \frac{1}{T} \sum_{t=1}^{T} r_t$$
where $r_t$ is the reward obtained at time $t$ following policy $\pi$.
...
0
votes
0answers
8 views
Markov Chain for Different Groups of Time Series
I would like to use (hidden) Markov Chain to predict $X[t+1]$ stock price. Historical data for top biggest 500 companies for last 10 years will be used for training.
The error would be sum of $...
2
votes
1answer
44 views
Metropolis-Hastings - interpreting the transition kernel: alpha*proposal
I thought I had great intuition and mathematical understanding of the Metropolis-Hastings algorithm, until closer inspection... as I started compiling my notes, I realized I do not understand the ...
1
vote
0answers
18 views
Running two MCMC chains in parallel while minimizing Kullback-Leibler divergence between both sample distributions
I want to sample from a distribution $p(X)$ with $X \in R^n$. However, I can only evaluate the likelihoods of $Z = AX$ and $Z = BX$ with $A,B \in R^{m \times n}$ and $m = n-1$.
Now my idea is to run ...
0
votes
0answers
6 views
Semi Markov Process for different condition states of a bridge
Following case: I have bridges in different condition states from 1 to 4 (good condition to worse). For every condition I estimated the weibull parameters form the data I got from an inspection (there ...
0
votes
0answers
9 views
Zero Sum problem with MDP formulation and the difference with minimax approach
Suppose that we have formulate a zero sum game with MDP and Ua(s) and Ub(s) are the utilities of A and B in the s state.
Suppose that all rewards and utilities are calculated from the A's point of ...
0
votes
0answers
8 views
Understanding Markov Chain source code in R [migrated]
The following source code is from a book. Comments are written by me to understand the code better.
...
1
vote
1answer
29 views
How can I compute $P(X_1 = 1|X_0 = 1)$ from the given data?
I know how to mathematically calculate the probability of various Markov Properties.
But, how can I calculate Markov probability from data?
Suppose, I have a Markov Chain as follows:
$S=\{1, 2\}$
...
2
votes
0answers
25 views
Understanding Approximate Dynamic Programming
I am trying to write a paper for my optimization class about Approximate Dynamic Programming. I found a few good papers but they all seem to dive straight into the material without talking about the ...
1
vote
1answer
24 views
Markov Blanket of two nodes?
I'm trying to solve a question and it has asked for (I feel like I'm confused by everything, would apprecaite some help, thanks),
the Markov blanket of {c,d}. From what I've read so far, Markov ...
0
votes
1answer
25 views
MCMC changing multiple parameters
What is the standard way of drawing multiple parameters for a MCMC run? Say I have 9 parameters, what is the most efficient method to get all parameters to explore their distributions properly?
1
vote
1answer
38 views
What happens if the observations are connected in a hidden Markov model (HMM)?
Suppose that we have an HMM with hidden variables $X_t$ and observed variables $Y_t$. Why do we always assume $p(Y_t|X_t)$? What happens if we have $p(Y_t|X_t, Y_{t-1})$? Is it because that wouldn't ...
2
votes
1answer
39 views
Manual simulation of Markov Chain in R
Consider the Markov chain with state space S = {1, 2}, transition matrix
and initial distribution α = (1/2, 1/2).
Simulate 5 steps of the Markov chain (that is, simulate X0, X1, . . ...
0
votes
1answer
27 views
Hidden Markov Model probability of producing a sequence
Suppose that we have two models for a 2-state HMM and both have two output symbols: $A$ and $B$.
Model 1:
Transition probabilities: $a_{11}=0.6$, $a_{12}=0.4$, $a_{21}=0.0$, $𝑎_{22}=1.0$.
Output ...
1
vote
1answer
56 views
For a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?
I am new to Markov Chains and using this concept in statistics.
For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?
If yes, how can I prove that?
0
votes
1answer
35 views
limit and stationary distribution of a Markov chain
Consider a Markov chain on the non-negative integers with transition
probabilities $1/2$ if $y=x+1$ and $1/2$ if $y=0$. Find $\lim_{n \to \infty} P(X_{n}=0)$.
Is this limit the same as the ...
0
votes
0answers
11 views
Countably infinite state space for Continuous Time Markov Chains
I have been unsuccessfully browsing the internet for resources related to Continuous Time Markov Chains with a countably infinite state space, for example $\mathbb{N}$.
Is there any developed theory ...
4
votes
2answers
81 views
Why does Judea Pearl call his causal graphs Markovian?
In his texts on causality, Judea Pearl always refers to the simplest graphs he uses, i.e. the acyclic graphs with independent confounders, as Markovian. I don't see why these graphs contain anything ...
0
votes
1answer
26 views
Compute the Limiting Distribution
Consider the transition matrix
$ P = \begin{bmatrix} 1-p&p\\ q&1-q \end{bmatrix} $
for general $2$-state Markov Chain $(0 \le p, q\le 1)$.
Find the limiting distribution ...
0
votes
0answers
12 views
Model selection with this model of a large number of components
I have a discrete time Markov Chain $\{X_n: n \in \mathbb{N}_0\}$ with unknown transition matrix $P \in \mathbb{R}^{M \times M}$ on the state space $\mathcal{S}_X = \{1,2, \dots, M\}$, with $M \geq 2$....
1
vote
1answer
22 views
How can I compute expected return time of a state in a Markov Chain?
I was watching a YouTube video regarding the calculation of expected return time of a Markov Chain.
https://www.youtube.com/watch?v=X_Ll0-Ytu7U&vl=en
I haven't understood the calculation of $m_{...
0
votes
0answers
14 views
Understanding the meaning of transition rates in a CTMC
I am reading the queueing theory volume 1 by Kleinrock. In the chapter on Continuous Time Markov Chain(CTMC), the author defines the infinitesimal generator, $Q(t)$ as having the following elements:
$...
1
vote
0answers
24 views
Markov property of manifest variables in longitudinal ESM
I am working on a longitudinal ESM model were the indicators are (highly) autocorrelated. This means that the classic cross-lagged models of panel data analysis cannot be used directly. I have ...
1
vote
0answers
19 views
Reversible markov chain
i have a question from a mid session exam. I was marked incorrect and I dont understand why. Question: Given a reversible markov chain P, with measure $\pi$. Show that $P^2$ is reversible.
Since P ...
0
votes
0answers
19 views
Markov Chain - Identifying the customers from model attribution results
I'm using Markov Chains to model a customers journey. My purpose is to attribute weights to channels, e.g. TV, FB, Billboards etc., for their relative importance.
For example, one arbitrary customer ...
0
votes
1answer
29 views
How to Show that a Distribution is a Stationary Distribution for Metropolis-Hastings? [closed]
For an Ising Model with a (2L+ 1) by (2L+ 1) square grid
of magnetic particles, show that $$\pi(\xi)=\frac{1}{Z_\beta}e^{\beta\sum_{x,y=x}{\xi_x\xi_y}}$$ Is indeed a stationary distribution for the ...
0
votes
0answers
37 views
How to use the Expectation Maximization (EM) algorithm for Part of Speech (POS) tagging?
I want to know how can we use the EM algorithm for Part of Speech (POS) tagging.
The data is a set of sentences Xs and their POS tags Ys i.e. a sentence X is a sequence of words $(X_1,X_2,\ldots, ...
0
votes
1answer
31 views
Kernel for a markov process?
Could anyone just explain to me what does it mean by mathematically, $P_n(x, dy)$ is the law of $X_n$ here in the page $46$.
https://statweb.stanford.edu/~cgates/PERSI/papers/iterate.pdf
Thanks for ...
1
vote
1answer
52 views
MCMC - target distribution, proposal distribution and likelihood function?
i have just started out in MCMC and I am not sure if I fully understand the concepts of MCMC with respect to the above terms. Let me try to explain that in my own words and include some thoughts / ...
5
votes
1answer
44 views
What does the distribution of samples from an MCMC method converge to without repeated samples?
Suppose I have an absolutely continuous distribution with density $f(x)$ and I use an mcmc sampler which has accept/reject step to sample from this distribution. In the final samples, there are some ...
0
votes
0answers
134 views
Markov chain: how to estimate the transition matrix? I don't have the underlying observations, just the sum by state and time
I have a matrix for data that (supposedly) follows a Markov process with an absorbing state; I have 3 possible states and 50 periods (discrete states, discrete time). Element [t,s] of the matrix tells ...
0
votes
1answer
29 views
What's the proper name for these chain structured PGMs?
I'm trying to find previous work that has dealt with this type of PGMs, but don't know what to call them:
a) "recurrent HMM"? $y_i$ are scalars and $x_i$ are discrete
b) "triangle HMM"? again, $y_i$ ...
3
votes
1answer
196 views
How does the Metropolis Algorithm “get off the ground”?
I'm thoroughly confused by the Metropolis Algorithm as defined in Casella and Berger's Statistical Inference. Namely, here's the definition (p.254):
Let $Y \sim f_Y(y)$ and $V \sim f_V(v)$, where $...
0
votes
1answer
23 views
Long run proportion of transitions in a Markov chain
Let $S$ be a set of states for a Markov chain and let $S^C$ be the remaining states. Explain the identity $$\sum_{i\in S}\sum_{j\in S^C}\pi_iP_{ij}=\sum_{i\in S^C}\sum_{j\in S}\pi_iP_{ij}$$
I know ...
1
vote
1answer
88 views
MCMC autocorrelation
I have a MCMC simulation that tries to fit a line to a linear set of data. The auto-correlation is very high for the slope parameter (~0.9), and low (~0.05) for the bias
What does a high auto-...
6
votes
1answer
86 views
How can we verify the intuition that in the RW-Metropolis-Hastings algorithm with Gaussian proposal too small and too large variances are bad choices
Let $d\in\mathbb N$ and consider the Random Walk Metropolis-Hastings algorithm with a Gaussian proposal kernel $Q$ such that $Q(x,\;\cdot\;)=\mathcal N_d(x,\sigma^2_dI_d)$ for all $x\in\mathbb R^d$.
...
0
votes
1answer
54 views
Relation between Uniform distribution, Metropolis Algorithm, and Symmetric Proposal Distribution
I am having some confusion over the Metropolis algorithm. Let $g(x|y)$ be our proposal distribution for the algorithm. For the Metropolis, $g$ must be symmetric (from Wikipedia). In the discrete case, ...
0
votes
1answer
16 views
Markov Chain Question/Notation Confusion
Show that if $(X_n)_{n \geq 0}$ is a discrete-time Markov chain with transition matrix $P$ and $Y_n = X_{kn}$, then $(Y_n)_{n \geq 0}$ is a Markov chain with transition matrix $P^k$.
I am a little ...
3
votes
1answer
93 views
How can we numerically compute the autocorrelation of a sample from a Markov chain generated by the Metropolis-Hastings algorithm?
Let $(X_n)_{n\in\mathbb N_0}$ denote a $\mathbb R^d$-valued Markov chain generated by the Metropolis-Hastings algorithm. Suppose I've run the algorithm on a computer and obtained a sample $x_0,\ldots,...
0
votes
0answers
11 views
Calculate Transition Probabilities Interest rate data
I came across a paper by Rodda (2004), who simulates interest rates with a Markov sequence. To simulate changes in the interest rates, they used the historical transition probabilities. Their ...
2
votes
0answers
60 views
Numerical examples proving and disproving the optimal scaling heuristic by Roberts et al
Let
$d\in\mathbb N$ with $d>1$
$\ell>0$
$\sigma_d^2:=\frac{\ell^2}{d-1}$
$f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$
$Q_d$ be a Markov kernel on $(\mathbb R^...
0
votes
0answers
46 views
Transition probability in Markov chain using a decision tree model
I wish to find a way to calculate the transition probabilities in my Markov chain model.
Let's say a customer has three products [A B C] and in this example I wish to know the transition probability ...
5
votes
1answer
95 views
Condition on the covariance matrix of a gaussian process needed to have the Markov property
Let suppose to have a realization $\mathbf{X}=(\mathbf{X}_1,\dots, \mathbf{X}_n)$, where $\mathbf{X}_i \in \mathcal{R}^d$, from a $d-$variate Gaussian process.
Let also suppose that $E(\mathbf{X}_i)= ...
0
votes
0answers
11 views
How to calculate nth step in the binary sequence?
I have a dataset with 499 observations for a single binary variable. The objective is to predict the next observation in the series.
Here is the dataset:
1 2 1 2 1 1 2 1 2 1 1 1 1 1 1 1 1 2 2 1 1 1 ...
5
votes
0answers
88 views
How does the celebrated result about the diffusion limit of the Random Walk Metroplis-Hastings algorithm help us to find the optimal scaling
Let
$d\in\mathbb N$ with $d>1$
$\ell>0$
$\sigma_d^2:=\frac{\ell^2}{d-1}$
$f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$
$Q_d$ be a Markov kernel on $(\mathbb R^...
0
votes
0answers
15 views
Sequence Prediction with noise / gap using markov models
I'm trying to understand if Markov models can account for a "noise event" when predicting the next item in a sequence.
For instance, if i have very frequently occurring (noise) event "F", can a ...
0
votes
0answers
38 views
What is the difference between homogeneous Markov chains and unhomogeneous Markov chains?
I learned that a Markov chain is a graph that describes how the state changes over time, and a homogeneous Markov chain is such a graph that its system dynamic doesn't change. Here the system dynamic ...
1
vote
0answers
49 views
Markov Models for time series prediction
I am student conducting an experiment with different models for time series prediction. In my experiment, I am going to use ARIMA, a Recurrent Neural Network, a Long-Short Term Memory network, and a ...
