# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

940 questions
Filter by
Sorted by
Tagged with
0answers
44 views
+50

### Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain

Let $(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces $f\in L^2(\lambda)$ $I$ be a finite nonempty set $\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
0answers
11 views

### How can we apply the rule of stationary distribution to the continuous case of Markov chain?

If the Markov chain converged then $$\pi = Q* \pi$$where $\pi$ is the posterior distribution and $Q$ is the transition distribution(it's a matrix in the discrete case). I tried to apply that on the ...
1answer
31 views

### Can we calculate the theoretical stationary distribution from a continuous Markov chain?

I have the transition distribution $p(X_{t+1}|X_t=x_t) = \text{N}(\phi x_t,1)$ where $−1<\phi<1$. Can we calculate the stationary distribution and its mean and variance? I know I can do that ...
1answer
22 views

### Hidden-Markov Model for Markov-Chain with Sequential Bernoulli State Sampling

Consider a finite discrete-time Markov chain whose state is sampled at the times determined by the outcome of a Bernoulli process. That is, in each time period I flip a biased coin. If it comes up as "...
1answer
23 views

0answers
49 views

### Make sense of plotting a transition matrix

I'm studying statistics and I'm trying to understand markov chain topic. I'm using the package "markovchain" in R to obtain the stationary distribution. From this transition matrix $M$: ...
0answers
10 views

### Approximate Value / Policy Iteration (Reinforcement Learning)

I am reading Markov Decision Processes in AI : about Approximate Dynamic Programming. Would you like to explain the rationale for introducing the API algorithm, how it compares to AVI ? How would ...
0answers
23 views

### Is there a reason why we should run the Metorpolis-Hastings algorithm with a target density approximating the density we're actually after?

Let $(E,\mathcal E,\lambda)$ be a measure space, $p:E\to[0,\infty)$ be $\mathcal E$-measurable with $$c:=\int p\:{\rm d}\lambda$$ and $$\mu:=\underbrace{\frac1cp}_{=:\:\tilde p}\lambda$$ denote the ...
0answers
10 views

### Can the interarrival times of a continuous time markov chain be distributed with 2 parameter (scale,location) exponential distributions?

I'm trying to model data with a time-homogenous CTMC with a number of states with corresponding constant transition rates $\lambda_{i}$ when I notice that much of the transition times from one state ...
0answers
6 views

### Markov chain transition Matrix for Ion channels in Python

I was working to code for a markov chain transition matrix for potassium channels. Potassium channels conists of 4 gates 1,2,3,4. the potassium ions antransition from 1->2 or 1->1, 2->1,2->2,2->3 etc. ...
1answer
15 views

### HMM rolling estimation different from batch estimation

I'm using the GuassianHMM from the python package hmmlearn and after fitting the hmm to the data the predictions that are done in one batch ...
0answers
45 views

0answers
12 views

### Markov Chain Monte Carlo for Decryption Purposes

I've been reading the "Markov Chain Monte Carlov Revolution" paper by Diaconis and was intrigued by his application to decryption. I've done some further reading and am now trying to work through ...
1answer
16 views

### What are the necessary qualifications or assumptions to say that a graph structure is a Markov Chain?

I have a graph structure and want to say it is a Markov Chain. But I am wondering what necessary assumptions or properties that my graph structure need to meet to be called a Markov Chain?
0answers
6 views

### What are the relationships among Markov Property, Stationarity, and Time Invariance

I am wondering if there is or are any relationship among those. I have understood Markov Property by reading Wikipedia, but it is still confusing to figure out if there is any relationship among those ...
0answers
24 views

### How to train a single discrete-time markov model?

I have a training set of sequences. I want to reach a discrete time Markov model (transition probability matrix). Is there a Bayesian way other than MLE to achieve this?
2answers
59 views

### Proof of (weak) consistency for an unbiased estimator

I want to prove a theorem stating: An unbiased estimator $\hat{\theta}$ of the unknown parameter $\theta$ is consistent if $V(\hat{\theta}_n$) $\to0$ for ${n\to\infty}$. I've tried using the ...
1answer
1k views

### Predicting the next value

I've recently read a blogpost where someone tries to predict the outcome of eurovision. Quoting from the summary of the site: Essentially, we can look at people’s voting preferences in the ...
1answer
53 views

### Spread of number of steps to reach absorbing state in markov chain

I know how to calculate the variance of the number of steps in an absorbing markov chain. However, I am not sure that the distribution of the number of steps is normal. Therefore I would like to ...
1answer
209 views

### Issue in graph construction

I have a symbolic representation of time series obtained from SAX toolbox. I was wondering if it is possible to construct a graph where each node represents a unique symbol and the edges represent the ...
1answer
210 views

### Question about marked poisson process

Let's say I have a Poisson point process on $\left[0,T\right]$ with rate $\lambda\left(t\right)=2t^2$. Suppose I attach a mark $m_t$ to each point $t$ of the process such that \$m_t\sim N\left(t,1\...
1answer
626 views

### Markov Chain and Removal Effect

Based on this article I'm trying to use within R the Channel Attribution package to leverage on the Markov Chain in order to attribute conversion between several marketing channel. On one point the ...
0answers
19 views

### How to find exact month of regime shifts in Markov switching model in R?

I'm looking into industry specific merger waves. I have used the Markov switching model (AR(0)) to identify periods with high M&A activity (merger wave periods) and periods with lower/normal M&...
1answer
329 views

### Measuring dependency of subsequent points from Markov chain

The question is about stimulating different type of species (coded 1-10) based on given species frequencies, and other parameters (eg. mean of normally distributed mass and ratio) using gibbs sampling....
1answer
273 views

### Can a Markov chain be approximated with an AR process?

In some MCMC literature/source code, a Markov chain is often approximated with an AR(1) process. There is some theory to suggest that such an approximation is somewhat valid for a finite state space, ...
0answers
196 views

### A Hidden Markov model with covariates in the transition probabilities

I would like to construct a Hidden Markov model with data about online customer journeys. A well-known concept related to the customer journey literature is the sales funnel. Consumers walk through ...