# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

372 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
306 views

### Reinforcement *Model* Learning

Classical reinforcement learning (Q- or Sarsa-Learning) can be extended with models of the environment. These models are usually transition tables that contain the probability of arriving at a ...
385 views

308 views

### Getting ROC curve from markov chain in r

I've been working on some attribution modeling in R, following the markov chain-based methodology described here. In order to compare the sensitivity and specificity trade-offs of using models of ...
295 views

114 views

### How can I determine the expected value / risk of ruin of game where probability changes dependent on state?

Consider the following game: You are given a random number generator which you can use to play a game. The object of the game is to reach the final tier after which you collect prize tokens. In ...
2k views

### How to understand Gibbs distribution

I have a graph model such as Following the Hammersley–Clifford theorem describes that Markov random fields exhibit a Gibbs distribution with an energy function as follows: P(x)=\frac {exp(-U(x))}{...
553 views

### How to estimate Markov chain transition probabilities with partially observed data?

Suppose that we have a time-homogeneous discrete-time Markov chain $(X_n)$. We want to estimate the transition probabilities $p_{ij} = \mathbb{P}[X_{n+1} = j \mid X_n = i]$. In the case when we have ...
954 views

### simulating birth death process with random numbers from negative binomial

I am trying to generate random deviates for the population size at time $t$ for a birth-death process with constant birth and death rates per individual and initial size $N_0 \gt 0$. For the simple ...
74 views

### Refining “good” mixing time estimate

Fix a Markov chain $\{ X_{t} \}_{t \in \mathbb{N}}$ with mixing time $\tau_{\mathrm{mix}}$. Assume that I know some finite bound on the mixing time $\tau_{\mathrm{mix}} < \tau < \infty$, and ...
153 views

### Markov Chain exercise in an exam

Suppose that $X_1, X_2, X_3...$ is a Markov chain with the following transition matrix: State: | 1 2 3 --+----------- 1 |0.2 0.4 0.4 2 |0.5 0.0 0.5 3 |0.6 0.3 0.1 (forgive my attempt at a ...
64 views

### Convergence time of a Markov chain

We know that a regular Markov chains converges to a unique matrix. The convergence time maybe finite or infinite. My interest is in the case where the convergence time is finite. How can we accurately ...
161 views

### Autoregressive Markov chain simulation and the likelihood ratio test for Markov property

I am trying to estimate a Markov chain of second order (Markov chain that fulfills $P[X_t|X_{t-1},X_{t-2}]=P[X_t|X_{t-1},X_{t-2},...,X_{t-p}]$) using an AR(2) process. Once I have simulated the ...
492 views

### How to find a conditional probability using copula-based Markov process?

I have a monthly time series of a water quality parameter. I used copula-based Markov process of C(Y(t), Y(t-1) and I forecasted the mean behavior of Yt by following equation: Now, I need to find ...
168 views

### Continuous time Markov chain forward equation

This is a homework question and I need suggestion how to approach it. We have given the transitions $\ i\rightarrow i+1$ with rate $\lambda(i)$ where $\ i \ge 1$ $\ i\rightarrow i-1$ with rate \$\...
202 views

### What is the expected number of steps before the spider exits? Markov Chain

A little spider lives in a rectangular box of which the sides are 3 and 4 cm long. It can only sit in one of the four corners marked with the numbers 1,2,3,4 (clockwise). Assume in Corner 2 there is a ...
418 views

### Estimating non-stationary Markov chain transition probabilities from data

I have some call centre outcome data: each person is called several times, and the result of the call is recorded as one of a few discrete outcomes ("no answer", "wrong number", "third party", "...
441 views

### How do sudden spikes affect hidden Markov models

I have some data that has two sudden spikes (almost like extreme masses or Dirac delta functions) within my time series data and I was wondering if that is a problem when building hidden Markov models....
534 views

### Are running mean plots good indicators for time series convergence?

I want to know whether a time series has converged. I looked around the MCMC literature (which requires doing something similar -- knowing whether a chain has converged) and found the following: http:...
1k views

### How to compare two matrices?

I am working on Markov transition matrices. I would like to find a statistical test to compare them. The first matrix is considered the population transition matrix and the second one is obtained by ...