# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

81 questions
Filter by
Sorted by
Tagged with
231 views

### Mars attack (probability to destroy $n$ spaceships with $k \cdot n$ missiles)

Suppose Earth has been attacked by $n$ Martian spaceships and suppose that we have $m=k \cdot n$ missiles to release against the $n$ spaceships. The probability to hit and destroy each spaceship by ...
8k views

### Solving the Kolmogorov forward equation for transition probabilities

Let $\lambda \mu > 0$ and let $X$ be a Markov chain on $\{1,2\}$ with generators $$Q = \begin{pmatrix} -\mu & \mu \\ \lambda & -\lambda \end{pmatrix}$$ Write down the forward equations ...
336 views

### Measuring dependency of subsequent points from Markov chain

The question is about stimulating different type of species (coded 1-10) based on given species frequencies, and other parameters (eg. mean of normally distributed mass and ratio) using gibbs sampling....
15k views

### Estimating Markov chain probabilities

What would be the common way of estimating MC transition matrix given the timeseries? Is there R function for doing that?
42k views

### Calculate Transition Matrix (Markov) in R

Is there a way in R (a built-in function) to calculate the transition matrix for a Markov Chain from a set of observations? For example, taking a data set like the following and calculate the first ...
2k views

### Can Hidden Markov Models be used to predict next observation?

I am reading up on Hidden Markov Models (HMMs) for my research and would like to know if it is applicable to the problem I wish to tackle. My problem is to detect/estimate the next value of a ...
12k views

### Resources for learning Markov chain and hidden Markov models

I am looking for resources (tutorials, textbooks, webcast, etc) to learn about Markov Chain and HMMs. My background is as a biologist, and I'm currently involved in a bioinformatics-related project. ...
13k views

### Markov Process about only depending on previous state

I would just like someone to confirm my understanding or if I'm missing something. The definition of a markov process says the next step depends on the current state only and no past states. So, let'...
7k views

27k views

### Intuitive explanation for periodicity in Markov chains

Can someone explain me in a intuitive way what the periodicity of a Markov chain is? It is defined as follows: For all states $i$ in $S$ $d_i$=gcd$\{n \in \mathbb{N} | p_{ii}^{(n)} > 0\} =1$ ...
7k views

### Converting 2nd order Markov chain to the 1st order equivalent

Given a 2nd order Markov chain where each state takes values in the set $\mathcal{X}=\{A,C,G,T\}$, such that all transition probabilities $p(x_t|x_{t-1},x_{t-2})$ are larger than zero, How to convert ...
7k views

### Problem in discrete valued time series forecasting

I have a temporally ordered discrete valued data. The only possible states for the data are: {1,2,3,4,5,6}. So the series is something like {1,2,3,5,6,4,3,5,2,......} I want to forecast the next value ...
299 views

### Are the mean of samples taken from Metropolis-Hastings MCMC normally distributed?

I've come across the following theorem while studying MCMC. It seems to suggest that the sample mean taken from the MCMC – the posterior marginal expectation – should be normally distributed, using ...
2k views

### Explanation regarding Gibbs Sampling

I am new to MCMC and reading a intro paper regarding Gibbs sampling. However, there are two parts in the paper I cannot understand and get stuck. The first part is equation 2.3 in page 168. It says ...
1k views

### Random matrices with constraints on row and column length

I need to generate random non-square matrices with $R$ rows and $C$ columns, elements randomly distributed with mean = 0, and constrained such that the length (L2 norm) of each row is $1$ and the ...
3k views

### How should one approch Project Euler problem 213 (“Flea Circus”)?

I would like to solve Project Euler 213 but don't know where to start because I'm a layperson in the field of Statistics, notice that an accurate answer is required so the Monte Carlo method won't ...
1k views

### Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are $\phi$-...
461 views

### What is the difference between Markov chain approximation and variational approximation?

I know they are two different approximation approaches to explicit models(which require approximation, that is transforming a non-optimization problem to an optimization problem to avoid the ...
2k views

### WinBUGS: Multiple definitions of a node

So this question is about the BUGS modeling language. So you either know it or have no clue. I'm a newbie to this so it's been driving me mad. I want to define a simple two-state hidden Markov model (...
211 views

### Markov chain getting stuck due to insufficient data samples

There is a lot of theory on Markov models and output generation out there, but I cannot locate any information on models getting stuck. I'm trying to create a model of a data set using a Markov model....
83 views

### Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain

Let $(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces $f\in L^2(\lambda)$ $I$ be a finite nonempty set $\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
882 views

### Deriving Transition Matrix of the Embedded Markov Chain given the generator matrix?

Full Problem: A continuous-time Markov chain has generator matrix $$Q= \begin{pmatrix} -1 & 1 & 0 \\ 1 & -2 & 1 \\ 2 & 2 & -4 \\ \end{pmatrix}$$ (i) ...
28k views

### Real-life examples of Markov Decision Processes

I've been watching a lot of tutorial videos and they are look the same. This one for example: https://www.youtube.com/watch?v=ip4iSMRW5X4 They explain states, actions and probabilities which are fine....
2k views

### Do we have a problem of “pity upvotes”?

I know, this may sound like it is off-topic, but hear me out. At Stack Overflow and here we get votes on posts, this is all stored in a tabular form. E.g.: post id voter id vote type ...
1k views

### Numeric solvers for stochastic differential equations in R: are there any?

I'm looking for a general, clean and fast (i.e. using C++ routines) R package for simulating paths from a non-homogeneous nonlinear diffusion like (1) using the Euler-Maruyama scheme, the Milstein ...