# Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

1,197 questions
Filter by
Sorted by
Tagged with
15 views

### Notation in Trust Region Policy Optimization by John Schulman et al

I am quite new to the area of reinforcement learning and find it hard to convice myself that the different notations used for reward function, state/action value function etc. coincide. Apparently I ...
14 views

### How is the clustering enhancement for this N-grams model useful practically?

The authors of this article An MDP-Based Recommender System use a predictive model based on N-grams on which they build their MDP model. One of the techniques introduced by the authors to make their ...
67 views

### Forecasting Future Values of Time Series [closed]

I am working with the R programming language. In particular, I am using "Markov Switching Models" for the purpose modelling more complicated dataset with varying degrees of volatility. For ...
13 views

### Contextual Bandit algorithms

I am trying to implement Contextual Bandit algorithms. I was going through the literature and the most recent algorithm I found was Taming the monster from 2014. I guess there surely must be more ...
25 views

44 views

### Simple Symmetric Random Walk on $\mathbb{Z}$ is null recurrent

Question: Consider a simple symmetric random walk on integers, where from every state $i$ you move to states $i-1$ and $i+1$ with probability half each. Show that this random walk is is null ...
38 views

### What are the requirements for a Markov chain to have a stationary distribution?

I read about Markov chains in quite a lot of different resources. However, I can't seem to find a consistent definition of what the requirements are for a Markov chain to have a stationary ...
28 views

### Understanding emission probability in HMM definition

This is rather basic question. I was going through Speech and Language Processing by Jurafsky and Martin. In the book, they define a Hidden Markov Model (HMM) as follows: An HMM is specified by the ...
153 views

### A farmer is growing a magical tree

This is not homework. It's a story I came up with to explain a statistical distribution I became interested in. If this is a known distribution, I'd love to be pointed in that direction. A farmer has ...
40 views

### In MCMC, can I accept proposals from another MCMC process without trying to approximate the proposal distribution?

I'm trying to sample a Markov chain which takes proposal from another Markov chain. Normally one would have a proposal distribution one could sample from. However, say in this case the proposals come ...
13 views

### How to verify if a graphical model has the markov property?

If I draw the computational graph of an HMM and an RNN from an architectural point of view they look very similar. The main difference is that an RNN gets some input $x$ and the HMM only operates on ...
20 views

### Calculating the degrees of freedom of a hidden Markov model (HMM)

I am curious if there is a straightforward explanation for calculating the degrees of freedom of a hidden Markov model (HMM). For example, take a simple HMM with a 1st-order Markov chain and 2 hidden ...
7 views

### MDP in Predictive Maintenance sample implementation

I am searching for a sample python implementation of Reinforcement Learning, Markov Decision Process in the domain of predictive maintenance. I have tried on my own, but either found sample related to ...
16 views

### What filter should I use for non gaussian distribution?

I have a process that measureing distance between 10-100mm and I currently measuring at 11-18mm with a fixed distance. I want to improve this measurement by adding a filter. Here is the distribution ...
91 views

24 views

### Markov property on non-stationary time-series

I am working with a dataset with several time-series, and I want to test the Markov property on all of them. However, some of them are non-stationary and following the definitions on the text "...
40 views

### Markov Process and a continuous PDF

I read an economics paper, and I got quite confused about the setting of the model: they assume that the labor productivity $\varepsilon$ (defined on a discrete state space) follows a Markov process. ...
17 views

### Expected value of time in absorbing Markov model

Assume we have a Markov model with an absorption state. We can find the expected value of absorbing time i.e., the expected value of the number of steps till we reach the absorbing state. Now assume ...
13 views

### Why does an estimated transition matrix have a positive entry for a non-allowed transition? (MSM package)

I have estimated a Markov model for panel data using the R package msm to calculate the transition matrix between states. This Markov model was estimated using an ...
53 views

### how can I scale a value beween 0 and 1? [closed]

I have written some AI software that calculates the behavior of the user. If the user behaves well, then the system will increase the user score by $a$. If the user behaves badly, the software ...
55 views

### Explicit solution to the invariant distribution of a Markov chain

Let $\{X_t\}_t$ be a discrete-time Markov chain with right stochastic transition matrix $P$ and a unique invariant distribution $\pi$. Let the state space be $\{1,\dots,S\}$. Is there an explicit ...
62 views

### Stationary distribution of a Markov chain with a random transition matrix

Consider a Markov chain $\{X_t\}$ on a finite state $\mathcal{S} = \{1,\dots, S\}$ space whose transition matrix $P$ is populated by elements of the form $$p_{ij} = P(X_{t+1} = j | X_t = i)$$ and we ...
22 views

### Determine order of events happening in discrete timespace

At $t=0$ I have a certain (active) population, lets say 1,000 customers. In each time step I expect a certain amount I have a probability of death of 1% and a prepayment rate of 5%. At the end of each ...
38 views

### Continuous Time Markov Chain - M/M/2 Queue throughput

First I find the stationary distribution of this problem by solving $\pi G=0$ to get $\pi \approx (0.20675105, 0.29535865, 0.05907173, 0.23628692, 0.1350211, 0.06751055)$. Using $\pi$, we can ...
15 views

### Strong stationarity and Markov property for an AR(1) process

Suppose I have an AR(1) process of the form $$X_t=\phi X_{t-1}+\epsilon_t,$$ where $\epsilon_t$ is a Gaussian white noise. Suppose that $X_t$ is weakly stationary check if $X_t$ is strongly ...
26 views

### Markov-Switching

I'm trying to make a Markov-Switching model in R for 3 regimes. I've got one data series with a regime of growth, decline, and sharp fall. ...
21 views

### How do I find a frequency of a sequence in a markov chain?

I have a 4x4 markov chain transition matrix T. Lets call the states A,B,C,D. I am looking for the large time expected frequency of a given sequence, say A-B-B-D. How exactly am I supposed to find this?...
89 views

### Question regarding positive recurrence

Suppose that $\{Y_n\}$ is an irreducible discrete time Markov chain with transition matrix $P = \{P_{i,j}\}_{i,j \in S}$. Let $\{Z_n\}$ be a discrete time Markov chain such that if it is in state $i$, ...
101 views

### How do I calculate the probability that some system will survive for some years before some event will occur?

I have a failure rate problem involving structural beams. The scenario is that a set of $n$ structural beams are supporting a weight of $W$ kilograms. For the purpose of simplification, we can assume ...
20 views

### Probability $X_n=k$ for a Markov Chain over the integers

Let $\{X_n\}$ be a Markov chain with state space $\mathcal{S}=\mathbb{Z}$, and $X_0=0,$ and $p_{0,1}=p_{0,-1}=1/2,$ and $p_{i,i+1}=1$ for all $i \ge 1,$ and $p_{i,i-1}=1$ for all $i\le -1.$ The ...
14 views

### Modified Renewal Process

I'm having trouble understanding how to deal with the modification to the queue. My thoughts were to use Poisson thinning to generate the infinitesimal matrix but that seems like a long shot. Any ...
15 views