# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,076 questions
Filter by
Sorted by
Tagged with
450 views

### Differences between Sampler, MonteCarlo, Metropolis-Hasting method, MCMC method and Fisher formalism

1) I make confusions about what we call a "sampler". From what I understand, a sampler allows to generate a distribution of points that follows a known PDF (probability distribution function)...
8 views

### What value does the deterministic policy take in an DPG objective function?

I have a doubt with the following paragraph, my doubts are quite basic: In the last 2 lines about deterministic policy, since it is deterministic, that is a single action would be taken given a state,...
59 views

### Metropolis-Hastings: target distribution with two modes; deterministic transformation

I'm trying to construct a Metropolis-Hastings algorithm to sample a target distribution $p(x)$ with two different and isolated modes. The example I'm working with is \begin{equation} p(x) = \frac{\...
418 views

11 views

### Mean covariate value in msm

I've been going through the manual of the R msm package for multi-state Markov models and have a question regarding how the default intensities are reported. The ...
717 views

### Detailed Balance for Hamiltonian Monte Carlo

I am trying to understand the detailed balance proof present in this paper: https://arxiv.org/abs/hep-lat/9208011v2 (page 4). My question: Why do we consider the volume of a neighborhood of points ...
7 views

### Correct way to present the definition a of Markov process of order $p$ for a vector process?

Usually when we define a Markov process of order $p$ for a univariate time-series $\{X_t\in\mathbb{R},t=1,2,\cdots\}$, the definition is presented as follows \begin{equation} P(X_t\leq x_t\mid x_1,\...
59 views

### Proving that the uniformly distributed stopping time doesnt have the memoryless property

Consider a chain which is not Markov that waits a time $T^{*}$ before leaving the current state, where $T^{*}$ has uniform distribution over the set of times $\{1, 2, 3, 4\}$ . I would like to show ...
I have a Markov chain with transition matrix $P$, with transition probabilities: $$p_{i,j}= \begin{cases} 1-d, & \text{if i=j \gt 0} \\[2ex] d , & \text{ if j=i-1 \gt 0} \\[2ex] (1- \... 0answers 5 views ### Bernoulli process with nonstationary probability Say we have a process X_t\vert P_t\sim \mathrm{Bin}(n,P_t) where X_t is observable but P_t is not. Also, the success probability P_t might vary over time and I don't assume some ... 0answers 16 views ### Car Driving Behavior: using non-instant transition with a Markov Transition Matrix I have a dataset on driving behavior/trips (workable example below), with information on the departure location, arrival location, duration of the trip and length of the trip. The states of the ... 1answer 38 views ### Struggling to understand the difference between a DTMC and a CTMC My textbook, Modeling and Analysis of Stochastic Systems, third edition, by Kulkarni, introduces continuous-time Markov chains (CTMCs) as follows: In Chapters 2, 3, and 4 we studied DTMCs. They arose ... 0answers 16 views ### Bellmans equation and existence of optimal policy for MDPs I'm trying to understand the proof of existence of an optimal policy from this question Why is there always at least one policy that is better than or equal to all other policies? by Lovelris. - ... 0answers 38 views ### Accuracy in Estimating Customer Lifetime Value using Markov Chain Model I've an online customer data which has the purchases made in every month and recency of the purchases information for 12 months. So data looks like below: ... 1answer 423 views ### The expected long run proportion of time the chain spends at a , given that it starts at c Consider the transition matrix: \begin{bmatrix} \frac{1}{5} & \frac{4}{5} & 0 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0 & 0 & 0 \\ \frac{1}{5} & \frac{1}{5} & \... 1answer 287 views ### Randomly generating transition probabilities for Markov chains I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ... 1answer 21 views ### Markov chains derivation for absorbing states I am given that the probability to reach a specific absorbing state s, from states 1, ..., M as a_1, \cdots, a_M, which are unique solutions to equations a_s = 1, a_0 = 0 for all absorbing ... 1answer 42 views ### Conversion of probabilities to rate [closed] I need to convert this matrix of weekly probabilities to annual probabilities. However, when converting it into annual rates and transforming them into probabilities does not give reasonable values ​​(... 1answer 156 views ### Finite state machine with gamma distributed waiting times I have a state machine with positive and negative inputs. The time between positive inputs follows a gamma distribution (X_+ \sim \Gamma(k_+, \theta_+)), and the time between negative inputs follows ... 1answer 21 views ### how to model random changes in line segments I have a discrete time stochastic process where an interval from 0 to L consists of smaller sub-segments, where the boundaries are always at integers, and L is some large integer. At each iteration, a ... 1answer 17 views ### Measuring quality of a timeseries model I want to create a model predicting whether an event will happen (let's say if Player A wins a match of ping-pong) conditional on a state (current score of the match). I have already developed a ... 2answers 37 views ### How does a Markov Chain converge to a distribution we don't know? As succinctly stated in this answer: it is possible to design a Markov chain with a stationary distribution equal to the posterior distribution, even though we don't know exactly what that ... 0answers 9 views ### What are the differences between Smoothed and filtered probabilities in Markov-Switching models? I am working with a MSM. I have noticed that almost all models present a plot that contains the smoothed and filtered probabilities, but I do not understand the differences between them. I understand ... 0answers 5 views ### Equal Limiting Distributions in a Markov Chain Implications I am studying Stochastic Models, and I came across the concept of limiting distributions, and I just wanted to make sure that I have understood it correctly. If the limiting probabilities are equal ... 1answer 20 views ### Arrival distribution of an M/M/1 queue Show that the arrivals A_n of an M/M/1 queue X with initial distribution \eta_i := \rho^{i-1}(1-\rho) (i \ge 1), where \rho is the traffic intensity, satisfy X_{A_n} \sim \ \eta. I ... 1answer 74 views ### how to solve this markov chain problem? This is a problem in the book of "introduction to stochastic process ". Any help to solve this problem ?? 2answers 29 views ### Markov Chain Converge then Unconverge? I have a MCMC algorithm for which I know the transition matrix T(x_{t+1},x_t). If the Markov chain has converged such that x_{t-1} \sim \pi, how do I show that x_i is marginally distributed ... 1answer 21 views ### Why Asymptotic Equipartition Property theorem proofs assume the source is memoryless? I do not understand the assumption 𝑋1,𝑋2,⋯ are i.i.d. ~p(x) in the AEP proofs I have seen. I have read some different sources for understanding the Asymptotic Equipartition Property. Using Cover &... 0answers 74 views ### Aggregate time-series forecast from individual probabilities I'm conceptualizing a methodology for a time series forecast but I lack the terminology and even the notation to learn more or even adequately describe it. Suppose I aim to forecast the aggregate ... 1answer 47 views ### \pi_i P^n_{i, j} = long-run proportion of time the chain is in i and will be in j after n transitions? I am currently studying the textbook Introduction to Probability Models by Sheldon M. Ross. Chapter 4.4 Long-Run Proportions and Limiting Probabilities says the following: Because \pi_i is the long-... 2answers 146 views ### How do you calculate the mean and variance of a random var with a distribution function that has a parameter with its own distribution function? I am busy with ruin theory.$$ S(t) = \sum_{i=1}^{N(t)} X_i $$S(t) is the aggregate claim size after t years, where X_i is the individual claim size (with mean and variance given) and N(t)... 0answers 47 views ### Combining probability and density with Bayes theorem I have prior density f(x), prior probabilities p(y=0), p(y=1) and two conditional densities:$$f(x|y=0) = \mathcal{N}(x, \mu_0, \sigma^2)f(x|y=1) = \mathcal{N}(x, \mu_1, \sigma^2)$$Where y \... 0answers 8 views ### What's the point in detailed balance? I'm not sure I understand how this pertains to a Markov chain? I've read that these equations prove the existence of a stationary distribution. However, don't I need to first find the stationary ... 1answer 21 views ### Steady state properties M/M/Inf/1/N queue Say I have a fixed population size of N individuals, each with exponential arrival times \lambda to an infinite number of queues with exponential service times \mu. I think the transition rate ... 2answers 90 views ### Monte Carlo Integration and pdfs? Let's say I have an un-normalized probability density function f(x), which is related to \xi via \xi = \frac{f}{c} I also have a sample set S = \{x_i\}_{i=1}^n \sim \xi which is sampled from ... 1answer 49 views ### What makes MCMC converge? Here is what I have learned about MCMC recently 1) We first propose a likelihood function that describes our problem (Binomial) 2) We define a conjugate prior (Beta) and posterior distribution (Beta-... 1answer 2k views ### Why is the optimal policy non-stationary in the case finite-horizon problems, whereas it is stationary in the case of infinite-horizon problems? I have difficulty understanding the meaning of stationary policy in the RL (MDP) setting. Specifically, let's assume stationary dynamics$$P(s_{t+1}=j|s_t=i,a) = P (s_{k+1}=j|s_k=i,a) \ \forall t,k,i,...
I found the following definition: "A probabilitly distribution $\pi = \{\pi_x\}_{x \in S}$ on the state space $S$ is called a stationary distribution for the Markov chain if for every $t > 0$,  ...