Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
1
vote
0answers
15 views

Notation in Trust Region Policy Optimization by John Schulman et al

I am quite new to the area of reinforcement learning and find it hard to convice myself that the different notations used for reward function, state/action value function etc. coincide. Apparently I ...
0
votes
0answers
14 views

How is the clustering enhancement for this N-grams model useful practically?

The authors of this article An MDP-Based Recommender System use a predictive model based on N-grams on which they build their MDP model. One of the techniques introduced by the authors to make their ...
1
vote
0answers
67 views

Forecasting Future Values of Time Series [closed]

I am working with the R programming language. In particular, I am using "Markov Switching Models" for the purpose modelling more complicated dataset with varying degrees of volatility. For ...
0
votes
0answers
13 views

Contextual Bandit algorithms

I am trying to implement Contextual Bandit algorithms. I was going through the literature and the most recent algorithm I found was Taming the monster from 2014. I guess there surely must be more ...
2
votes
1answer
25 views

Rejection sampling: Can the proposal distribution be the prior?

Suppose I have a target distribution $\pi(\theta|x) \propto P(x|\theta)P_{\theta}(\theta)$ (e.g. the unnormalized posterior). I would like to use rejection sampling to obtain many samples $\{\theta_i\}...
0
votes
1answer
25 views

Dynamically adjusting parameters of Markov chain

I am using a Metropolis algorithm to generate samples from a complicated (high-dimensional) probability distribution. As is common, the proposed updates depend on some "step size" parameter $...
0
votes
0answers
44 views

Simple Symmetric Random Walk on $\mathbb{Z}$ is null recurrent

Question: Consider a simple symmetric random walk on integers, where from every state $i$ you move to states $i-1$ and $ i+1$ with probability half each. Show that this random walk is is null ...
0
votes
1answer
38 views

What are the requirements for a Markov chain to have a stationary distribution?

I read about Markov chains in quite a lot of different resources. However, I can't seem to find a consistent definition of what the requirements are for a Markov chain to have a stationary ...
1
vote
1answer
28 views

Understanding emission probability in HMM definition

This is rather basic question. I was going through Speech and Language Processing by Jurafsky and Martin. In the book, they define a Hidden Markov Model (HMM) as follows: An HMM is specified by the ...
11
votes
2answers
153 views

A farmer is growing a magical tree

This is not homework. It's a story I came up with to explain a statistical distribution I became interested in. If this is a known distribution, I'd love to be pointed in that direction. A farmer has ...
1
vote
1answer
40 views

In MCMC, can I accept proposals from another MCMC process without trying to approximate the proposal distribution?

I'm trying to sample a Markov chain which takes proposal from another Markov chain. Normally one would have a proposal distribution one could sample from. However, say in this case the proposals come ...
0
votes
0answers
13 views

How to verify if a graphical model has the markov property?

If I draw the computational graph of an HMM and an RNN from an architectural point of view they look very similar. The main difference is that an RNN gets some input $x$ and the HMM only operates on ...
0
votes
1answer
20 views

Calculating the degrees of freedom of a hidden Markov model (HMM)

I am curious if there is a straightforward explanation for calculating the degrees of freedom of a hidden Markov model (HMM). For example, take a simple HMM with a 1st-order Markov chain and 2 hidden ...
0
votes
0answers
7 views

MDP in Predictive Maintenance sample implementation

I am searching for a sample python implementation of Reinforcement Learning, Markov Decision Process in the domain of predictive maintenance. I have tried on my own, but either found sample related to ...
0
votes
0answers
16 views

What filter should I use for non gaussian distribution?

I have a process that measureing distance between 10-100mm and I currently measuring at 11-18mm with a fixed distance. I want to improve this measurement by adding a filter. Here is the distribution ...
3
votes
0answers
91 views

Split a two dimensional continuous time Markov chain into two independent ones?

Let's say we have a two dimensional MC defined on the state space $\mathbb{N}\times \mathbb{N}$ evolving as below: $(i,j) \rightarrow (i,j+1)$ with rate $\lambda$ for all $i,j$. $(i,j) \rightarrow (i-...
0
votes
0answers
10 views

Proof independence using Markov networks

Let $f$ be a probability density function. Also, let $V_1, ..., V_n$ be vertices in the Markov network graph. Prove if $f(V_1| V_2, V_3, ..., V_n) = f(V_1| V_3, ..., V_n)$ then there is no edge ...
0
votes
0answers
22 views

Confused about notation of conditional entropy and cross-entropy in Markov chain example

I was reading an IT paper where this Markov chain appeared: $X \to Y|X \to X|Y$. I do not recall conditional r.v. By the context of the paper, $X \sim P(X)$ and $Y|X$ and $X|Y$ are the results of ...
1
vote
0answers
298 views

How would this Markov chain problem be solved?

I have the following problem: Bob is a salesman. Each week, Bob will either make no money, make a small amount of money, or make a large amount of money. And if Bob makes money, he will either make \$...
0
votes
0answers
24 views

Markov property on non-stationary time-series

I am working with a dataset with several time-series, and I want to test the Markov property on all of them. However, some of them are non-stationary and following the definitions on the text "...
1
vote
1answer
40 views

Markov Process and a continuous PDF

I read an economics paper, and I got quite confused about the setting of the model: they assume that the labor productivity $\varepsilon$ (defined on a discrete state space) follows a Markov process. ...
0
votes
0answers
17 views

Expected value of time in absorbing Markov model

Assume we have a Markov model with an absorption state. We can find the expected value of absorbing time i.e., the expected value of the number of steps till we reach the absorbing state. Now assume ...
0
votes
0answers
13 views

Why does an estimated transition matrix have a positive entry for a non-allowed transition? (MSM package)

I have estimated a Markov model for panel data using the R package msm to calculate the transition matrix between states. This Markov model was estimated using an ...
0
votes
3answers
53 views

how can I scale a value beween 0 and 1? [closed]

I have written some AI software that calculates the behavior of the user. If the user behaves well, then the system will increase the user score by $a$. If the user behaves badly, the software ...
2
votes
1answer
55 views

Explicit solution to the invariant distribution of a Markov chain

Let $\{X_t\}_t$ be a discrete-time Markov chain with right stochastic transition matrix $P$ and a unique invariant distribution $\pi$. Let the state space be $\{1,\dots,S\}$. Is there an explicit ...
2
votes
1answer
62 views

Stationary distribution of a Markov chain with a random transition matrix

Consider a Markov chain $\{X_t\}$ on a finite state $\mathcal{S} = \{1,\dots, S\}$ space whose transition matrix $P$ is populated by elements of the form $$ p_{ij} = P(X_{t+1} = j | X_t = i)$$ and we ...
1
vote
0answers
22 views

Determine order of events happening in discrete timespace

At $t=0$ I have a certain (active) population, lets say 1,000 customers. In each time step I expect a certain amount I have a probability of death of 1% and a prepayment rate of 5%. At the end of each ...
-1
votes
1answer
38 views

Continuous Time Markov Chain - M/M/2 Queue throughput

First I find the stationary distribution of this problem by solving $\pi G=0$ to get $\pi \approx (0.20675105, 0.29535865, 0.05907173, 0.23628692, 0.1350211, 0.06751055)$. Using $\pi$, we can ...
0
votes
0answers
15 views

Strong stationarity and Markov property for an AR(1) process

Suppose I have an AR(1) process of the form $$X_t=\phi X_{t-1}+\epsilon_t,$$ where $\epsilon_t$ is a Gaussian white noise. Suppose that $X_t$ is weakly stationary check if $X_t$ is strongly ...
0
votes
0answers
26 views

Markov-Switching

I'm trying to make a Markov-Switching model in R for 3 regimes. I've got one data series with a regime of growth, decline, and sharp fall. ...
0
votes
0answers
21 views

How do I find a frequency of a sequence in a markov chain?

I have a 4x4 markov chain transition matrix T. Lets call the states A,B,C,D. I am looking for the large time expected frequency of a given sequence, say A-B-B-D. How exactly am I supposed to find this?...
3
votes
1answer
89 views

Question regarding positive recurrence

Suppose that $\{Y_n\}$ is an irreducible discrete time Markov chain with transition matrix $P = \{P_{i,j}\}_{i,j \in S}$. Let $\{Z_n\}$ be a discrete time Markov chain such that if it is in state $i$, ...
4
votes
1answer
101 views

How do I calculate the probability that some system will survive for some years before some event will occur?

I have a failure rate problem involving structural beams. The scenario is that a set of $n$ structural beams are supporting a weight of $W$ kilograms. For the purpose of simplification, we can assume ...
0
votes
0answers
20 views

Probability $X_n=k$ for a Markov Chain over the integers

Let $\{X_n\}$ be a Markov chain with state space $\mathcal{S}=\mathbb{Z}$, and $X_0=0,$ and $p_{0,1}=p_{0,-1}=1/2,$ and $p_{i,i+1}=1$ for all $i \ge 1,$ and $p_{i,i-1}=1$ for all $i\le -1.$ The ...
0
votes
0answers
14 views

Modified Renewal Process

I'm having trouble understanding how to deal with the modification to the queue. My thoughts were to use Poisson thinning to generate the infinitesimal matrix but that seems like a long shot. Any ...
0
votes
0answers
15 views

Solving for the parameters of a discrete markov state model

I have some observations $y_t \in \mathbb{R}_+, t\in 0 \dots n$, and I'd like to fit the following model: a vector of initial states $p_0 \in \mathbb{R}^s_+$, transition probabilities $K \in \mathbb{R}...
0
votes
0answers
14 views

Using Markov Chains to predict market share for prescription drugs

I wish to understand if the approach I am researching holds up to analytical rigor. I am planning to use historical data of 4 prescription drugs to forecast the market share of the same 4 drugs some ...
1
vote
0answers
14 views

LR test for overlapping return data using bootstrap

I wish to test a null hypothesis as in Christoffersen (1998) to see whether a sequence of Value-at-Risk forecasts $Q_t(p) \in \mathcal{F}_t$ possesses correct conditional coverage. Here $p \in [0,1]$ ...
0
votes
0answers
9 views

derivation for piecewise deterministic Markov process

I was reading a formulation of piecewise-deterministic Markov process $\Pi_t$, $ t \in \mathbb{R}_+$. In particular $\Pi_t$ is defined as $\Pi_t = P (X_t | \mathcal{F}_{\lfloor t/ \Delta \rfloor}) $ ...
0
votes
0answers
16 views

Deciding on sequence prediction approach

I have a space of 128 states expressed with binary bitstrings. The transition probabilities among these states are known to me, as follows: ...
0
votes
0answers
22 views

Event-based markov chain with observed covariates at non-constant time intervals

I am working on a project where I need to identify the traffic state at intersections. More specifically, I want to classify the situation in 5 states: undersaturation (where the traffic needs to stop ...
0
votes
0answers
47 views

When does a stationary distribution for a CTMC not exist?

Recalling that a Continuous-time Markov Chain (CTMC) is defined by its generator matrix $Q$, when does a stationary distribution for a CTMC not exist? Or when does there not exist a probability vector ...
1
vote
0answers
28 views

Markov Chain for Amino Acid Sequence

I was wondering if it is possible to apply Markov Chain to this amino acid sequence "ddvlsldeddddsdyncgednd" to find the probability of this sequence forming in this particular positional ...
0
votes
0answers
19 views

Proving positive recurrence for a finite state space DTMC

Let $$P = \begin{pmatrix} 0.3 & 0.5 & 0 & 0 & 0 & 0.2 \\ 0 & 0.5 & 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 0.3 & 0 & 0 &...
1
vote
2answers
100 views

Markov Chains - Can we set the Initial distribution as $X_{1}$ instead of $X_{0}$ to calculate a non-conditional probability?

Had a small question when I was going through Ross – Probability Models. Given a discrete-time Markov chain $\{X_{n}: n=1,2,3 \ldots\}$ with a state space $\mathcal{S} = \{0,1\}$, and an initial ...
0
votes
0answers
15 views

Rankings transition matrix

I have heard of ratings transition matrices in finance, where it is possible to derive the probability of moving from one rating to another in a Markov chain process. If I have N individuals who are ...
1
vote
1answer
25 views

What does it mean for a Markov CHAIN to be recurrent (not just a state)?

There are many resources offering equivalent definitions of recurrence for a state in a Markov Chain - for example, state $x$ is recurrent if, starting in state $x$ you will eventually return to state ...
0
votes
1answer
28 views

Question regarding limiting probability

Does there exist an irreducible discrete time Markov chain such that all limiting probabilities exist, i.e. $\lim\limits_{k \to \infty} \mathbb{P}(X_k = i)$ exists for all states $i$, and $\sum_{i} \...
0
votes
0answers
29 views

How to calculate var(var) in time series

I try to calculate var(var) with Markov Process in time series by Bartlett(1946, P28) https://www.jstor.org/stable/2983611?seq=2#metadata_info_tab_contents , I use quadratic form calculate,but I didn'...
0
votes
0answers
8 views

When to use a markov switching model?

When can you use a markov switching model? For example if I was looking at the effect of inflation on bank liquidity, could it be used here?

1
2 3 4 5
24