Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

376 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
307 views

Reinforcement *Model* Learning

Classical reinforcement learning (Q- or Sarsa-Learning) can be extended with models of the environment. These models are usually transition tables that contain the probability of arriving at a ...
389 views

317 views

Getting ROC curve from markov chain in r

I've been working on some attribution modeling in R, following the markov chain-based methodology described here. In order to compare the sensitivity and specificity trade-offs of using models of ...
299 views

114 views

How can I determine the expected value / risk of ruin of game where probability changes dependent on state?

Consider the following game: You are given a random number generator which you can use to play a game. The object of the game is to reach the final tier after which you collect prize tokens. In ...
2k views

How to understand Gibbs distribution

I have a graph model such as Following the Hammersley–Clifford theorem describes that Markov random fields exhibit a Gibbs distribution with an energy function as follows: P(x)=\frac {exp(-U(x))}{...
562 views

How to estimate Markov chain transition probabilities with partially observed data?

Suppose that we have a time-homogeneous discrete-time Markov chain $(X_n)$. We want to estimate the transition probabilities $p_{ij} = \mathbb{P}[X_{n+1} = j \mid X_n = i]$. In the case when we have ...
974 views

simulating birth death process with random numbers from negative binomial

I am trying to generate random deviates for the population size at time $t$ for a birth-death process with constant birth and death rates per individual and initial size $N_0 \gt 0$. For the simple ...
74 views

Refining “good” mixing time estimate

Fix a Markov chain $\{ X_{t} \}_{t \in \mathbb{N}}$ with mixing time $\tau_{\mathrm{mix}}$. Assume that I know some finite bound on the mixing time $\tau_{\mathrm{mix}} < \tau < \infty$, and ...
156 views

Markov Chain exercise in an exam

Suppose that $X_1, X_2, X_3...$ is a Markov chain with the following transition matrix: State: | 1 2 3 --+----------- 1 |0.2 0.4 0.4 2 |0.5 0.0 0.5 3 |0.6 0.3 0.1 (forgive my attempt at a ...
64 views

Convergence time of a Markov chain

We know that a regular Markov chains converges to a unique matrix. The convergence time maybe finite or infinite. My interest is in the case where the convergence time is finite. How can we accurately ...
163 views

Autoregressive Markov chain simulation and the likelihood ratio test for Markov property

I am trying to estimate a Markov chain of second order (Markov chain that fulfills $P[X_t|X_{t-1},X_{t-2}]=P[X_t|X_{t-1},X_{t-2},...,X_{t-p}]$) using an AR(2) process. Once I have simulated the ...
494 views

How to find a conditional probability using copula-based Markov process?

I have a monthly time series of a water quality parameter. I used copula-based Markov process of C(Y(t), Y(t-1) and I forecasted the mean behavior of Yt by following equation: Now, I need to find ...
168 views

Continuous time Markov chain forward equation

This is a homework question and I need suggestion how to approach it. We have given the transitions $\ i\rightarrow i+1$ with rate $\lambda(i)$ where $\ i \ge 1$ $\ i\rightarrow i-1$ with rate \$\...
211 views

What is the expected number of steps before the spider exits? Markov Chain

A little spider lives in a rectangular box of which the sides are 3 and 4 cm long. It can only sit in one of the four corners marked with the numbers 1,2,3,4 (clockwise). Assume in Corner 2 there is a ...
422 views

Estimating non-stationary Markov chain transition probabilities from data

I have some call centre outcome data: each person is called several times, and the result of the call is recorded as one of a few discrete outcomes ("no answer", "wrong number", "third party", "...
442 views

How do sudden spikes affect hidden Markov models

I have some data that has two sudden spikes (almost like extreme masses or Dirac delta functions) within my time series data and I was wondering if that is a problem when building hidden Markov models....
547 views

Are running mean plots good indicators for time series convergence?

I want to know whether a time series has converged. I looked around the MCMC literature (which requires doing something similar -- knowing whether a chain has converged) and found the following: http:...