Questions tagged [markov-process]
A stochastic process with the property that the future is conditionally independent of the past, given the present.
1,076
questions
0
votes
0answers
9 views
What value does the deterministic policy take in an DPG objective function?
I have a doubt with the following paragraph, my doubts are quite basic:
In the last 2 lines about deterministic policy, since it is deterministic, that is a single action would be taken given a state,...
2
votes
1answer
65 views
Metropolis-Hastings: target distribution with two modes; deterministic transformation
I'm trying to construct a Metropolis-Hastings algorithm to sample a target distribution $p(x)$ with two different and isolated modes. The example I'm working with is
\begin{equation}
p(x) = \frac{\...
0
votes
0answers
13 views
Does HMM training data need observed states?
I have been trying to study HMMs and have had some differences in understanding them with a colleague with whom I am working on a project. I would really appreciate some clarification.
From what I ...
0
votes
0answers
6 views
Transition probablities vs transition rate
What would you describe and differentiate between Transition rate and probabilities intuitively in accordance with Transition probablity matrix as well as markov chains and HSD models.
0
votes
2answers
16 views
How to calculate the probability Matrix (Alpha) for Regular Markov chains
Pardon me for being a novice here. In the image attached, eq 3.1 represents the transition matrix (it's pretty clear). I am not able to comprehend the eq 3.2, alpha*P = alpha, as well as the further ...
0
votes
1answer
43 views
Why is this code is not implementing Markov Chain
My MOOC course says that this code is not implementing Markov Chain
...
0
votes
0answers
10 views
Encoding a time series with varying time differences as an image
There are methods to encode a time series into an 'image' i.e a matrix of scalar values. Some methods include recurrence plots, gramian angular field and markov transition field. Most methods assume ...
2
votes
1answer
28 views
How to define number of states in reinforcement learning
I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
1
vote
1answer
44 views
Are RNNs Markovian?
On the one hand, one can argue that they are since "the hidden layer is simply [derived from] the last hidden state and current input". On the other hand, the whole point of RNNs is that &...
2
votes
2answers
34 views
How is the law of total probability used here?
Consider a Markov chain $\{X_n, n = 0, 1, \dots\}$.
The probability of going from one state $i$ to state $j$ in two steps is $p_{ij}^2 = P(X_2 = j | X_0 = i)$.
Then by the law of total probability ...
3
votes
1answer
33 views
Can I still call a chain a Markov Chain if it is not ergodic, and can I still use it for prediction?
Currently, I am using a Markov Chain to build a predictive model. I have done some research on the Internet, and found that a Markov Chain has a stationary distribution followed by ergodic condition.
...
1
vote
0answers
11 views
Mean covariate value in msm
I've been going through the manual of the R msm package for multi-state Markov models and have a question regarding how the default intensities are reported. The ...
0
votes
0answers
7 views
Correct way to present the definition a of Markov process of order $p$ for a vector process?
Usually when we define a Markov process of order $p$ for a univariate time-series $\{X_t\in\mathbb{R},t=1,2,\cdots\}$, the definition is presented as follows
\begin{equation}
P(X_t\leq x_t\mid x_1,\...
0
votes
0answers
5 views
Bernoulli process with nonstationary probability
Say we have a process $X_t\vert P_t\sim \mathrm{Bin}(n,P_t)$ where $X_t$ is observable but $P_t$ is not. Also, the success probability $P_t$ might vary over time and I don't assume some ...
1
vote
0answers
16 views
Car Driving Behavior: using non-instant transition with a Markov Transition Matrix
I have a dataset on driving behavior/trips (workable example below), with information on the departure location, arrival location, duration of the trip and length of the trip. The states of the ...
0
votes
1answer
38 views
Struggling to understand the difference between a DTMC and a CTMC
My textbook, Modeling and Analysis of Stochastic Systems, third edition, by Kulkarni, introduces continuous-time Markov chains (CTMCs) as follows:
In Chapters 2, 3, and 4 we studied DTMCs. They arose ...
0
votes
0answers
16 views
Bellmans equation and existence of optimal policy for MDPs
I'm trying to understand the proof of existence of an optimal policy from this question
Why is there always at least one policy that is better than or equal to all other policies?
by Lovelris.
-
...
1
vote
1answer
42 views
Conversion of probabilities to rate [closed]
I need to convert this matrix of weekly probabilities to annual probabilities. However, when converting it into annual rates and transforming them into probabilities does not give reasonable values āā(...
0
votes
0answers
38 views
Accuracy in Estimating Customer Lifetime Value using Markov Chain Model
I've an online customer data which has the purchases made in every month and recency of the purchases information for 12 months. So data looks like below:
...
2
votes
1answer
19 views
Measuring quality of a timeseries model
I want to create a model predicting whether an event will happen (let's say if Player A wins a match of ping-pong) conditional on a state (current score of the match).
I have already developed a ...
2
votes
2answers
37 views
How does a Markov Chain converge to a distribution we don't know?
As succinctly stated in this answer:
it is possible to design a Markov chain with a stationary distribution equal to the posterior distribution, even though we don't know exactly what that ...
0
votes
0answers
9 views
What are the differences between Smoothed and filtered probabilities in Markov-Switching models?
I am working with a MSM. I have noticed that almost all models present a plot that contains the smoothed and filtered probabilities, but I do not understand the differences between them. I understand ...
0
votes
0answers
5 views
Equal Limiting Distributions in a Markov Chain Implications
I am studying Stochastic Models, and I came across the concept of limiting distributions, and I just wanted to make sure that I have understood it correctly.
If the limiting probabilities are equal ...
0
votes
1answer
21 views
Markov chains derivation for absorbing states
I am given that the probability to reach a specific absorbing state $s$, from states 1, ..., M as $a_1, \cdots, a_M$, which are unique solutions to equations $a_s = 1$, $a_0 = 0$ for all absorbing ...
1
vote
2answers
29 views
Markov Chain Converge then Unconverge?
I have a MCMC algorithm for which I know the transition matrix $T(x_{t+1},x_t)$. If the Markov chain has converged such that $x_{t-1} \sim \pi$, how do I show that $x_i$ is marginally distributed ...
0
votes
0answers
74 views
Aggregate time-series forecast from individual probabilities
I'm conceptualizing a methodology for a time series forecast but I lack the terminology and even the notation to learn more or even adequately describe it.
Suppose I aim to forecast the aggregate ...
0
votes
1answer
47 views
$\pi_i P^n_{i, j} =$ long-run proportion of time the chain is in $i$ and will be in $j$ after $n$ transitions?
I am currently studying the textbook Introduction to Probability Models by Sheldon M. Ross. Chapter 4.4 Long-Run Proportions and Limiting Probabilities says the following:
Because $\pi_i$ is the long-...
0
votes
0answers
8 views
What's the point in detailed balance?
I'm not sure I understand how this pertains to a Markov chain?
I've read that these equations prove the existence of a stationary distribution. However, don't I need to first find the stationary ...
2
votes
0answers
49 views
Combining probability and density with Bayes theorem
I have prior density $f(x)$, prior probabilities $p(y=0), p(y=1)$ and two conditional densities:
$$f(x|y=0) = \mathcal{N}(x, \mu_0, \sigma^2)$$
$$f(x|y=1) = \mathcal{N}(x, \mu_1, \sigma^2)$$
Where $y \...
0
votes
1answer
21 views
Steady state properties M/M/Inf/1/N queue
Say I have a fixed population size of $N$ individuals, each with exponential arrival times $\lambda$ to an infinite number of queues with exponential service times $\mu$.
I think the transition rate ...
1
vote
1answer
21 views
how to model random changes in line segments
I have a discrete time stochastic process where an interval from 0 to L consists of smaller sub-segments, where the boundaries are always at integers, and L is some large integer. At each iteration, a ...
1
vote
2answers
90 views
Monte Carlo Integration and pdfs?
Let's say I have an un-normalized probability density function $f(x)$, which is related to $\xi$ via $\xi = \frac{f}{c}$
I also have a sample set $S = \{x_i\}_{i=1}^n \sim \xi$ which is sampled from ...
0
votes
1answer
21 views
Why Asymptotic Equipartition Property theorem proofs assume the source is memoryless?
I do not understand the assumption š1,š2,āÆ are i.i.d. ~p(x) in the AEP proofs I have seen. I have read some different sources for understanding the Asymptotic Equipartition Property. Using Cover &...
1
vote
0answers
11 views
How do I work with a master SIR equation?
Using the definitions here and a standard Markov chain, I arrived at the following master equation:
$$\partial_t P_{S,I,R}(t) = (\beta/N)(I-1)(S+1)P_{S+1,I-1,R}(t) + \gamma(I+1)P_{S,I+1,R-1}(t)-(\beta ...
0
votes
0answers
14 views
Where is the “Markov property” in Markov Random Fields? Are Markov Random Fields stochastic processes at all?
I am learning about Markom Random Fields. As I understand them they are constituted of some Random variables $X_{i}$ and some potential functions $\phi(X_s)$ where $X_s$ is a set of some random ...
1
vote
1answer
36 views
Probability of reaching goal in n steps?
Consider a simple grid-like setup where an agent starts at the first state (s0) and has to reach the absorbing state (G):
| s0 | s1 | G |
Also, when it tries to ...
0
votes
0answers
14 views
Is a non-iid process necessarily a Markov Process?
This may be a silly question, but I've been wondering if a a random variable is said to follow a non-iid process, does that necessarily imply that it follows a Markov process?
1
vote
1answer
53 views
Show that an irreducible Markov Chain is always reversible
Suppose we have a Markov chain $X$, with transition probabilities $T_{ij} = P(X_{n+1}=i\mid X_{n}=j )$.
Let $(Y_{1}, Y_{2}, .., Y_{N})$ represent the reversed Markov chain of $X$, such that $Y_{n} = ...
1
vote
0answers
28 views
How to compare the chi-square distance values obtained from comparing different histogram pairs?
I have a data sequence that is generated from a computer simulation (discrete event simulation). It is a simple sequence of a single state value, e.g., 10 11 12 11 12 13 ...15. It contains N number of ...
2
votes
1answer
20 views
Arrival distribution of an M/M/1 queue
Show that the arrivals $A_n$ of an M/M/1 queue $X$ with initial distribution $\eta_i := \rho^{i-1}(1-\rho)$ ($i \ge 1$), where $\rho$ is the traffic intensity, satisfy $X_{A_n} \sim \ \eta$.
I ...
0
votes
0answers
5 views
How to test the Markov property and homogeneity of Markov chain using the likelihood ration of Markov chain in R?
First, I have done my analysisĀ using the firstĀ order Markov chain.
How to test the Markov property and homogeneity of Markov chain using the likelihood ration of Markov chain in R?
Second, I have ...
2
votes
1answer
44 views
Is this even possible?
How is this possible:
If $P (Z|Y, X) = P(Z|Y)$ AND $P(X|Y,Z)= P(X|Y)$
How can these two be equal: $$P (Z|Y,X) = P(X|Y,Z)$$
0
votes
0answers
6 views
Hidden Markov Model with Autoregressive Emissions
So far, all standard HMM implementations I've seen assume some variation of a Gaussian Mixture (GMM) as their emission model. It can of course only have a single mixture component which reduces it to ...
-1
votes
1answer
49 views
What makes MCMC converge?
Here is what I have learned about MCMC recently
1) We first propose a likelihood function that describes our problem (Binomial)
2) We define a conjugate prior (Beta) and posterior distribution (Beta-...
3
votes
1answer
127 views
Conditional expectation for maximum function
I have a discrete-time Markov chain queuing problem.
Packets (computer packets, that is) arrive in the intervals. $A_n$ denotes the number of arrivals in the interval $(n - 1, n)$, where $n \ge 1$, ...
0
votes
1answer
52 views
The question is related to stochastic process and the Markov chains
This is the complete question.
Jim is currently living in Scranton. Each year that he lives in Scranton, he has a
probability of 1/2 of staying in Scranton the next year. Otherwise, he has an equally
...
0
votes
0answers
34 views
Building an hierarchical model
I am trying to understand hierarchic Bayesian models.
I am following the example http://www.openbugs.net/Examples/Pumps.html.
The exercise I am trying to complete relates to the same problem, with the ...
2
votes
1answer
72 views
Proving (or disproving) a property for Markov Chains
I have to prove or disprove the following: Let $X_n$ be a Markov Chain on state space $S = \{1,2,3,4,5,6\}$. Then
$$P(X_2 = 6 | X_1 \in \{3,4\}, X_0 = 2) = P(X_2 = 6 | X_1 \in \{3,4\}).$$
This ...
0
votes
1answer
27 views
generate a realization from a transition matrix
Consider a markov chain of 4 states $\{S_1, S_2, S_3, S_4\}$ described by the transition matrix
$$
A = \begin{bmatrix}
.25 & .20 & .25 & .30 \\
.20 & .30 & .25 & .30 \\
....
0
votes
0answers
15 views
Markov model for multiple cases
I was wondering if the markovchain r package can handle multiple cases with unequal length of sequences. If so, how to perform that in r? If not, any other markov packages in r can do that? Many ...
