Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
9
votes
2answers
450 views

Differences between Sampler, MonteCarlo, Metropolis-Hasting method, MCMC method and Fisher formalism

1) I make confusions about what we call a "sampler". From what I understand, a sampler allows to generate a distribution of points that follows a known PDF (probability distribution function)...
0
votes
0answers
8 views

What value does the deterministic policy take in an DPG objective function?

I have a doubt with the following paragraph, my doubts are quite basic: In the last 2 lines about deterministic policy, since it is deterministic, that is a single action would be taken given a state,...
2
votes
1answer
59 views

Metropolis-Hastings: target distribution with two modes; deterministic transformation

I'm trying to construct a Metropolis-Hastings algorithm to sample a target distribution $p(x)$ with two different and isolated modes. The example I'm working with is \begin{equation} p(x) = \frac{\...
8
votes
1answer
418 views

Markov Switching Forecast. How can I derive this?

Consider the autoregressive model, $\left[ \begin{array}{l} y^{\ast}_t\\ x_t^{\ast} \end{array} \right] = \left[ \begin{array}{l} a_{11}\\ a_{21} \end{array} \begin{array}{l} a_{12}\\ a_{...
0
votes
1answer
306 views

Multi-step ahead prediction using multivariate Markov model

I need to perform multi-step ahead prediction using multivariate Markov model. Do we need to update the transition matrix after each prediction or use the same. How can we update it based on ...
0
votes
2answers
15 views

How to calculate the probability Matrix (Alpha) for Regular Markov chains

Pardon me for being a novice here. In the image attached, eq 3.1 represents the transition matrix (it's pretty clear). I am not able to comprehend the eq 3.2, alpha*P = alpha, as well as the further ...
0
votes
0answers
12 views

Does HMM training data need observed states?

I have been trying to study HMMs and have had some differences in understanding them with a colleague with whom I am working on a project. I would really appreciate some clarification. From what I ...
0
votes
0answers
6 views

Transition probablities vs transition rate

What would you describe and differentiate between Transition rate and probabilities intuitively in accordance with Transition probablity matrix as well as markov chains and HSD models.
0
votes
1answer
42 views

Why is this code is not implementing Markov Chain

My MOOC course says that this code is not implementing Markov Chain ...
0
votes
0answers
10 views

Encoding a time series with varying time differences as an image

There are methods to encode a time series into an 'image' i.e a matrix of scalar values. Some methods include recurrence plots, gramian angular field and markov transition field. Most methods assume ...
2
votes
1answer
27 views

How to define number of states in reinforcement learning

I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
1
vote
1answer
44 views

Are RNNs Markovian?

On the one hand, one can argue that they are since "the hidden layer is simply [derived from] the last hidden state and current input". On the other hand, the whole point of RNNs is that &...
2
votes
2answers
34 views

How is the law of total probability used here?

Consider a Markov chain $\{X_n, n = 0, 1, \dots\}$. The probability of going from one state $i$ to state $j$ in two steps is $p_{ij}^2 = P(X_2 = j | X_0 = i)$. Then by the law of total probability ...
3
votes
1answer
224 views

Grid search methods for posterior distribution approximation

I'm reading the book "Bayesian Analysis with Python" and the author provides some python code designed to show the grid search method of obtaining an approximate posterior distribution for the classic ...
3
votes
1answer
33 views

Can I still call a chain a Markov Chain if it is not ergodic, and can I still use it for prediction?

Currently, I am using a Markov Chain to build a predictive model. I have done some research on the Internet, and found that a Markov Chain has a stationary distribution followed by ergodic condition. ...
5
votes
1answer
116 views

How does the celebrated result about the diffusion limit of the Random Walk Metroplis-Hastings algorithm help us to find the optimal scaling

Let $d\in\mathbb N$ with $d>1$ $\ell>0$ $\sigma_d^2:=\frac{\ell^2}{d-1}$ $f\in C^2(\mathbb R)$ be positive with $$\int f(x)\:{\rm d}x=1$$ and $g:=\ln f$ $Q_d$ be a Markov kernel on $(\mathbb R^...
1
vote
0answers
11 views

Mean covariate value in msm

I've been going through the manual of the R msm package for multi-state Markov models and have a question regarding how the default intensities are reported. The ...
4
votes
0answers
717 views

Detailed Balance for Hamiltonian Monte Carlo

I am trying to understand the detailed balance proof present in this paper: https://arxiv.org/abs/hep-lat/9208011v2 (page 4). My question: Why do we consider the volume of a neighborhood of points ...
0
votes
0answers
7 views

Correct way to present the definition a of Markov process of order $p$ for a vector process?

Usually when we define a Markov process of order $p$ for a univariate time-series $\{X_t\in\mathbb{R},t=1,2,\cdots\}$, the definition is presented as follows \begin{equation} P(X_t\leq x_t\mid x_1,\...
2
votes
1answer
59 views

Proving that the uniformly distributed stopping time doesnt have the memoryless property

Consider a chain which is not Markov that waits a time $T^{*}$ before leaving the current state, where $T^{*}$ has uniform distribution over the set of times $\{1, 2, 3, 4\}$ . I would like to show ...
3
votes
0answers
58 views

Markov chain with stopping times

I have a Markov chain with transition matrix $P$, with transition probabilities: $$p_{i,j}= \begin{cases} 1-d, & \text{if $i=j \gt 0$} \\[2ex] d , & \text{ if $j=i-1 \gt 0$} \\[2ex] (1- \...
0
votes
0answers
5 views

Bernoulli process with nonstationary probability

Say we have a process $X_t\vert P_t\sim \mathrm{Bin}(n,P_t)$ where $X_t$ is observable but $P_t$ is not. Also, the success probability $P_t$ might vary over time and I don't assume some ...
1
vote
0answers
16 views

Car Driving Behavior: using non-instant transition with a Markov Transition Matrix

I have a dataset on driving behavior/trips (workable example below), with information on the departure location, arrival location, duration of the trip and length of the trip. The states of the ...
0
votes
1answer
38 views

Struggling to understand the difference between a DTMC and a CTMC

My textbook, Modeling and Analysis of Stochastic Systems, third edition, by Kulkarni, introduces continuous-time Markov chains (CTMCs) as follows: In Chapters 2, 3, and 4 we studied DTMCs. They arose ...
0
votes
0answers
16 views

Bellmans equation and existence of optimal policy for MDPs

I'm trying to understand the proof of existence of an optimal policy from this question Why is there always at least one policy that is better than or equal to all other policies? by Lovelris. - ...
0
votes
0answers
38 views

Accuracy in Estimating Customer Lifetime Value using Markov Chain Model

I've an online customer data which has the purchases made in every month and recency of the purchases information for 12 months. So data looks like below: ...
5
votes
1answer
423 views

The expected long run proportion of time the chain spends at $a$ , given that it starts at $c$

Consider the transition matrix: $\begin{bmatrix} \frac{1}{5} & \frac{4}{5} & 0 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 \\ \frac{1}{5} & \frac{1}{5} & \...
1
vote
1answer
287 views

Randomly generating transition probabilities for Markov chains

I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ...
0
votes
1answer
21 views

Markov chains derivation for absorbing states

I am given that the probability to reach a specific absorbing state $s$, from states 1, ..., M as $a_1, \cdots, a_M$, which are unique solutions to equations $a_s = 1$, $a_0 = 0$ for all absorbing ...
1
vote
1answer
42 views

Conversion of probabilities to rate [closed]

I need to convert this matrix of weekly probabilities to annual probabilities. However, when converting it into annual rates and transforming them into probabilities does not give reasonable values ​​(...
7
votes
1answer
156 views

Finite state machine with gamma distributed waiting times

I have a state machine with positive and negative inputs. The time between positive inputs follows a gamma distribution ($X_+ \sim \Gamma(k_+, \theta_+)$), and the time between negative inputs follows ...
1
vote
1answer
21 views

how to model random changes in line segments

I have a discrete time stochastic process where an interval from 0 to L consists of smaller sub-segments, where the boundaries are always at integers, and L is some large integer. At each iteration, a ...
2
votes
1answer
17 views

Measuring quality of a timeseries model

I want to create a model predicting whether an event will happen (let's say if Player A wins a match of ping-pong) conditional on a state (current score of the match). I have already developed a ...
2
votes
2answers
37 views

How does a Markov Chain converge to a distribution we don't know?

As succinctly stated in this answer: it is possible to design a Markov chain with a stationary distribution equal to the posterior distribution, even though we don't know exactly what that ...
0
votes
0answers
9 views

What are the differences between Smoothed and filtered probabilities in Markov-Switching models?

I am working with a MSM. I have noticed that almost all models present a plot that contains the smoothed and filtered probabilities, but I do not understand the differences between them. I understand ...
0
votes
0answers
5 views

Equal Limiting Distributions in a Markov Chain Implications

I am studying Stochastic Models, and I came across the concept of limiting distributions, and I just wanted to make sure that I have understood it correctly. If the limiting probabilities are equal ...
2
votes
1answer
20 views

Arrival distribution of an M/M/1 queue

Show that the arrivals $A_n$ of an M/M/1 queue $X$ with initial distribution $\eta_i := \rho^{i-1}(1-\rho)$ ($i \ge 1$), where $\rho$ is the traffic intensity, satisfy $X_{A_n} \sim \ \eta$. I ...
0
votes
1answer
74 views

how to solve this markov chain problem?

This is a problem in the book of "introduction to stochastic process ". Any help to solve this problem ??
1
vote
2answers
29 views

Markov Chain Converge then Unconverge?

I have a MCMC algorithm for which I know the transition matrix $T(x_{t+1},x_t)$. If the Markov chain has converged such that $x_{t-1} \sim \pi$, how do I show that $x_i$ is marginally distributed ...
0
votes
1answer
21 views

Why Asymptotic Equipartition Property theorem proofs assume the source is memoryless?

I do not understand the assumption 𝑋1,𝑋2,β‹― are i.i.d. ~p(x) in the AEP proofs I have seen. I have read some different sources for understanding the Asymptotic Equipartition Property. Using Cover &...
0
votes
0answers
74 views

Aggregate time-series forecast from individual probabilities

I'm conceptualizing a methodology for a time series forecast but I lack the terminology and even the notation to learn more or even adequately describe it. Suppose I aim to forecast the aggregate ...
0
votes
1answer
47 views

$\pi_i P^n_{i, j} =$ long-run proportion of time the chain is in $i$ and will be in $j$ after $n$ transitions?

I am currently studying the textbook Introduction to Probability Models by Sheldon M. Ross. Chapter 4.4 Long-Run Proportions and Limiting Probabilities says the following: Because $\pi_i$ is the long-...
0
votes
2answers
146 views

How do you calculate the mean and variance of a random var with a distribution function that has a parameter with its own distribution function?

I am busy with ruin theory. $$ S(t) = \sum_{i=1}^{N(t)} X_i $$ $S(t)$ is the aggregate claim size after $t$ years, where $X_i$ is the individual claim size (with mean and variance given) and $N(t)$...
2
votes
0answers
47 views

Combining probability and density with Bayes theorem

I have prior density $f(x)$, prior probabilities $p(y=0), p(y=1)$ and two conditional densities: $$f(x|y=0) = \mathcal{N}(x, \mu_0, \sigma^2)$$ $$f(x|y=1) = \mathcal{N}(x, \mu_1, \sigma^2)$$ Where $y \...
0
votes
0answers
8 views

What's the point in detailed balance?

I'm not sure I understand how this pertains to a Markov chain? I've read that these equations prove the existence of a stationary distribution. However, don't I need to first find the stationary ...
0
votes
1answer
21 views

Steady state properties M/M/Inf/1/N queue

Say I have a fixed population size of $N$ individuals, each with exponential arrival times $\lambda$ to an infinite number of queues with exponential service times $\mu$. I think the transition rate ...
1
vote
2answers
90 views

Monte Carlo Integration and pdfs?

Let's say I have an un-normalized probability density function $f(x)$, which is related to $\xi$ via $\xi = \frac{f}{c}$ I also have a sample set $S = \{x_i\}_{i=1}^n \sim \xi$ which is sampled from ...
-1
votes
1answer
49 views

What makes MCMC converge?

Here is what I have learned about MCMC recently 1) We first propose a likelihood function that describes our problem (Binomial) 2) We define a conjugate prior (Beta) and posterior distribution (Beta-...
4
votes
1answer
2k views

Why is the optimal policy non-stationary in the case finite-horizon problems, whereas it is stationary in the case of infinite-horizon problems?

I have difficulty understanding the meaning of stationary policy in the RL (MDP) setting. Specifically, let's assume stationary dynamics $$P(s_{t+1}=j|s_t=i,a) = P (s_{k+1}=j|s_k=i,a) \ \forall t,k,i,...
2
votes
2answers
1k views

Definition of stationary distribution in continuous time markov chains

I found the following definition: "A probabilitly distribution $\pi = \{\pi_x\}_{x \in S}$ on the state space $S$ is called a stationary distribution for the Markov chain if for every $t > 0$, $$ ...

1
2 3 4 5
…
22