Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,197
questions
1
vote
0answers
15 views
Notation in Trust Region Policy Optimization by John Schulman et al
I am quite new to the area of reinforcement learning and find it hard to convice myself that the different notations used for reward function, state/action value function etc. coincide.
Apparently I ...
0
votes
0answers
14 views
How is the clustering enhancement for this N-grams model useful practically?
The authors of this article An MDP-Based Recommender System use a predictive model based on N-grams on which they build their MDP model. One of the techniques introduced by the authors to make their ...
1
vote
0answers
67 views
Forecasting Future Values of Time Series [closed]
I am working with the R programming language. In particular, I am using "Markov Switching Models" for the purpose modelling more complicated dataset with varying degrees of volatility. For ...
0
votes
0answers
13 views
Contextual Bandit algorithms
I am trying to implement Contextual Bandit algorithms. I was going through the literature and the most recent algorithm I found was Taming the monster from 2014. I guess there surely must be more ...
2
votes
1answer
25 views
Rejection sampling: Can the proposal distribution be the prior?
Suppose I have a target distribution $\pi(\theta|x) \propto P(x|\theta)P_{\theta}(\theta)$ (e.g. the unnormalized posterior). I would like to use rejection sampling to obtain many samples $\{\theta_i\}...
0
votes
1answer
25 views
Dynamically adjusting parameters of Markov chain
I am using a Metropolis algorithm to generate samples from a complicated (high-dimensional) probability distribution. As is common, the proposed updates depend on some "step size" parameter $...
0
votes
0answers
44 views
Simple Symmetric Random Walk on $\mathbb{Z}$ is null recurrent
Question: Consider a simple symmetric random walk on integers, where from every state $i$ you move to states $i-1$ and $ i+1$ with probability half each. Show that this random walk is is null ...
0
votes
1answer
38 views
What are the requirements for a Markov chain to have a stationary distribution?
I read about Markov chains in quite a lot of different resources.
However, I can't seem to find a consistent definition of what the requirements are for a Markov chain to have a stationary ...
1
vote
1answer
28 views
Understanding emission probability in HMM definition
This is rather basic question. I was going through Speech and Language Processing by Jurafsky and Martin. In the book, they define a Hidden Markov Model (HMM) as follows:
An HMM is specified by the ...
11
votes
2answers
153 views
A farmer is growing a magical tree
This is not homework. It's a story I came up with to explain a statistical distribution I became interested in. If this is a known distribution, I'd love to be pointed in that direction.
A farmer has ...
1
vote
1answer
40 views
In MCMC, can I accept proposals from another MCMC process without trying to approximate the proposal distribution?
I'm trying to sample a Markov chain which takes proposal from another Markov chain. Normally one would have a proposal distribution one could sample from. However, say in this case the proposals come ...
0
votes
0answers
13 views
How to verify if a graphical model has the markov property?
If I draw the computational graph of an HMM and an RNN from an architectural point of view they look very similar. The main difference is that an RNN gets some input $x$ and the HMM only operates on ...
0
votes
1answer
20 views
Calculating the degrees of freedom of a hidden Markov model (HMM)
I am curious if there is a straightforward explanation for calculating the degrees of freedom of a hidden Markov model (HMM).
For example, take a simple HMM with a 1st-order Markov chain and 2 hidden ...
0
votes
0answers
7 views
MDP in Predictive Maintenance sample implementation
I am searching for a sample python implementation of Reinforcement Learning, Markov Decision Process in the domain of predictive maintenance.
I have tried on my own, but either found sample related to ...
0
votes
0answers
16 views
What filter should I use for non gaussian distribution?
I have a process that measureing distance between 10-100mm and I currently measuring at 11-18mm with a fixed distance. I want to improve this measurement by adding a filter.
Here is the distribution ...
3
votes
0answers
91 views
Split a two dimensional continuous time Markov chain into two independent ones?
Let's say we have a two dimensional MC defined on the state space $\mathbb{N}\times \mathbb{N}$ evolving as below:
$(i,j) \rightarrow (i,j+1)$ with rate $\lambda$ for all $i,j$.
$(i,j) \rightarrow (i-...
0
votes
0answers
10 views
Proof independence using Markov networks
Let $f$ be a probability density function. Also, let $V_1, ..., V_n$ be vertices in the Markov network graph.
Prove if $f(V_1| V_2, V_3, ..., V_n) = f(V_1| V_3, ..., V_n)$ then there is no edge ...
0
votes
0answers
22 views
Confused about notation of conditional entropy and cross-entropy in Markov chain example
I was reading an IT paper where this Markov chain appeared: $X \to Y|X \to X|Y$.
I do not recall conditional r.v. By the context of the paper, $X \sim P(X)$ and $Y|X$ and $X|Y$ are the results of ...
1
vote
0answers
298 views
How would this Markov chain problem be solved?
I have the following problem:
Bob is a salesman. Each week, Bob will either make no money, make a small amount of money, or make a large amount of money. And if Bob makes money, he will either make \$...
0
votes
0answers
24 views
Markov property on non-stationary time-series
I am working with a dataset with several time-series, and I want to test the Markov property on all of them. However, some of them are non-stationary and following the definitions on the text "...
1
vote
1answer
40 views
Markov Process and a continuous PDF
I read an economics paper, and I got quite confused about the setting of the model: they assume that the labor productivity $\varepsilon$ (defined on a discrete state space) follows a Markov process. ...
0
votes
0answers
17 views
Expected value of time in absorbing Markov model
Assume we have a Markov model with an absorption state. We can find the expected value of absorbing time i.e., the expected value of the number of steps till we reach the absorbing state. Now assume ...
0
votes
0answers
13 views
Why does an estimated transition matrix have a positive entry for a non-allowed transition? (MSM package)
I have estimated a Markov model for panel data using the R package msm to calculate the transition matrix between states. This Markov model was estimated using an ...
0
votes
3answers
53 views
how can I scale a value beween 0 and 1? [closed]
I have written some AI software that calculates the behavior of the user. If the user behaves well, then the system will increase the user score by $a$. If the user behaves badly, the software ...
2
votes
1answer
55 views
Explicit solution to the invariant distribution of a Markov chain
Let $\{X_t\}_t$ be a discrete-time Markov chain with right stochastic transition matrix $P$ and a unique invariant distribution $\pi$. Let the state space be $\{1,\dots,S\}$. Is there an explicit ...
2
votes
1answer
62 views
Stationary distribution of a Markov chain with a random transition matrix
Consider a Markov chain $\{X_t\}$ on a finite state $\mathcal{S} = \{1,\dots, S\}$ space whose transition matrix $P$ is populated by elements of the form
$$ p_{ij} = P(X_{t+1} = j | X_t = i)$$
and we ...
1
vote
0answers
22 views
Determine order of events happening in discrete timespace
At $t=0$ I have a certain (active) population, lets say 1,000 customers. In each time step I expect a certain amount I have a probability of death of 1% and a prepayment rate of 5%. At the end of each ...
-1
votes
1answer
38 views
Continuous Time Markov Chain - M/M/2 Queue throughput
First I find the stationary distribution of this problem by solving $\pi G=0$ to get $\pi \approx (0.20675105, 0.29535865, 0.05907173, 0.23628692, 0.1350211, 0.06751055)$. Using $\pi$, we can ...
0
votes
0answers
15 views
Strong stationarity and Markov property for an AR(1) process
Suppose I have an AR(1) process of the form
$$X_t=\phi X_{t-1}+\epsilon_t,$$
where $\epsilon_t$ is a Gaussian white noise. Suppose that $X_t$ is weakly stationary
check if $X_t$ is strongly ...
0
votes
0answers
26 views
Markov-Switching
I'm trying to make a Markov-Switching model in R for 3 regimes. I've got one data series with a regime of growth, decline, and sharp fall.
...
0
votes
0answers
21 views
How do I find a frequency of a sequence in a markov chain?
I have a 4x4 markov chain transition matrix T. Lets call the states A,B,C,D. I am looking for the large time expected frequency of a given sequence, say A-B-B-D. How exactly am I supposed to find this?...
3
votes
1answer
89 views
Question regarding positive recurrence
Suppose that $\{Y_n\}$ is an irreducible discrete time Markov chain with transition matrix $P = \{P_{i,j}\}_{i,j \in S}$. Let $\{Z_n\}$ be a discrete time Markov chain such that if it is in state $i$, ...
4
votes
1answer
101 views
How do I calculate the probability that some system will survive for some years before some event will occur?
I have a failure rate problem involving structural beams. The scenario is that a set of $n$ structural beams are supporting a weight of $W$ kilograms. For the purpose of simplification, we can assume ...
0
votes
0answers
20 views
Probability $X_n=k$ for a Markov Chain over the integers
Let $\{X_n\}$ be a Markov chain with state space $\mathcal{S}=\mathbb{Z}$, and $X_0=0,$ and $p_{0,1}=p_{0,-1}=1/2,$ and $p_{i,i+1}=1$ for all $i \ge 1,$ and $p_{i,i-1}=1$ for all $i\le -1.$ The ...
0
votes
0answers
14 views
Modified Renewal Process
I'm having trouble understanding how to deal with the modification to the queue. My thoughts were to use Poisson thinning to generate the infinitesimal matrix but that seems like a long shot. Any ...
0
votes
0answers
15 views
Solving for the parameters of a discrete markov state model
I have some observations $y_t \in \mathbb{R}_+, t\in 0 \dots n$, and I'd like to fit the following model: a vector of initial states $p_0 \in \mathbb{R}^s_+$, transition probabilities $K \in \mathbb{R}...
0
votes
0answers
14 views
Using Markov Chains to predict market share for prescription drugs
I wish to understand if the approach I am researching holds up to analytical rigor.
I am planning to use historical data of 4 prescription drugs to forecast the market share of the same 4 drugs some ...
1
vote
0answers
14 views
LR test for overlapping return data using bootstrap
I wish to test a null hypothesis as in Christoffersen (1998) to see whether a sequence of Value-at-Risk forecasts $Q_t(p) \in \mathcal{F}_t$ possesses correct conditional coverage. Here $p \in [0,1]$ ...
0
votes
0answers
9 views
derivation for piecewise deterministic Markov process
I was reading a formulation of piecewise-deterministic Markov process $\Pi_t$, $ t \in \mathbb{R}_+$. In particular $\Pi_t$ is defined as $\Pi_t = P (X_t | \mathcal{F}_{\lfloor t/ \Delta \rfloor}) $ ...
0
votes
0answers
16 views
Deciding on sequence prediction approach
I have a space of 128 states expressed with binary bitstrings. The transition probabilities among these states are known to me, as follows:
...
0
votes
0answers
22 views
Event-based markov chain with observed covariates at non-constant time intervals
I am working on a project where I need to identify the traffic state at intersections. More specifically, I want to classify the situation in 5 states: undersaturation (where the traffic needs to stop ...
0
votes
0answers
47 views
When does a stationary distribution for a CTMC not exist?
Recalling that a Continuous-time Markov Chain (CTMC) is defined by its generator matrix $Q$, when does a stationary distribution for a CTMC not exist? Or when does there not exist a probability vector ...
1
vote
0answers
28 views
Markov Chain for Amino Acid Sequence
I was wondering if it is possible to apply Markov Chain to this amino acid sequence "ddvlsldeddddsdyncgednd" to find the probability of this sequence forming in this particular positional ...
0
votes
0answers
19 views
Proving positive recurrence for a finite state space DTMC
Let $$P =
\begin{pmatrix}
0.3 & 0.5 & 0 & 0 & 0 & 0.2 \\
0 & 0.5 & 0 & 0.5 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 & 0 \\
0 & 0.3 & 0 & 0 &...
1
vote
2answers
100 views
Markov Chains - Can we set the Initial distribution as $X_{1}$ instead of $X_{0}$ to calculate a non-conditional probability?
Had a small question when I was going through Ross – Probability Models. Given a discrete-time Markov chain
$\{X_{n}: n=1,2,3 \ldots\}$ with a state space $\mathcal{S} = \{0,1\}$, and an initial ...
0
votes
0answers
15 views
Rankings transition matrix
I have heard of ratings transition matrices in finance, where it is possible to derive the probability of moving from one rating to another in a Markov chain process. If I have N individuals who are ...
1
vote
1answer
25 views
What does it mean for a Markov CHAIN to be recurrent (not just a state)?
There are many resources offering equivalent definitions of recurrence for a state in a Markov Chain - for example, state $x$ is recurrent if, starting in state $x$ you will eventually return to state ...
0
votes
1answer
28 views
Question regarding limiting probability
Does there exist an irreducible discrete time Markov chain such that all limiting probabilities exist, i.e. $\lim\limits_{k \to \infty} \mathbb{P}(X_k = i)$ exists for all states $i$, and $\sum_{i} \...
0
votes
0answers
29 views
How to calculate var(var) in time series
I try to calculate var(var) with Markov Process in time series by Bartlett(1946, P28) https://www.jstor.org/stable/2983611?seq=2#metadata_info_tab_contents , I use quadratic form calculate,but I didn'...
0
votes
0answers
8 views
When to use a markov switching model?
When can you use a markov switching model? For example if I was looking at the effect of inflation on bank liquidity, could it be used here?
