The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
1
vote
0answers
9 views

Central limit theorem for a Metropolis-Hastings estimator

Let $\kappa$ denote the transition kernel of the Markov chain $(X_n)_{n\in\mathbb N_0}$ generated by the Metropolis-Hastings algorithm with proposal kernel $Q$ and target distribution $\mu$$^1$ and $(...
2
votes
1answer
106 views

Point process model diagnostic: Nearest-Neighbor Distance Distribution or Pair Correlation Function?

I have a point pattern which is clearly inhomogeneous. Furthermore, the inhomogeneity has two components: a large scale effect and a local scale effect. I have constructed a Markov point process model ...
0
votes
0answers
10 views

Does this problem satisfy markov properties to be modeled as HMM?

I want to model a chemical reaction network which is defined by a stoichiometric matrix $\nu^{s\times m} $ where $s$ is the number of participating species and $m$ the number of chemical reactions. If ...
4
votes
4answers
7k views

Aperiodicity in markov chain

given this transition matrix of markov chain 1/2 1/4 1/4 0 1/2 1/2 1 0 0 which represents transition matrix of states a,b,c. a has probability of 1/2 ...
0
votes
1answer
56 views

Could someone help me to understand the Metropolis-Hastings algorithm for discrete Markov Chains?

Metropolis-Hastings Algorithm Assume the Markov chain is in some state $X_{n} = i$. Let $\textbf{H}$ be the transition matrix for any irreducible Markov chain on the state space. We generate $X_{n+1}$...
0
votes
0answers
17 views

the convergence speed for a Markov chain

For a metropolis hastings algorithm, suppose that the stationary distribution is defined as the Gibbs Boltzmann distribution $\pi_T(x)= \frac{1}{Z_T}e^{-\frac{V(x)}{T} }$ where $Z_T = \sum_{y\in V} e^{...
0
votes
1answer
132 views

Probabilistic user behavior markov models on web

I am considering the following probabilistic Markov model of actions of a user on the results page of a search engine. The user examines the first result, with a probability $A$ he is satisfied with ...
5
votes
1answer
318 views

Getting ROC curve from markov chain in r

I've been working on some attribution modeling in R, following the markov chain-based methodology described here. In order to compare the sensitivity and specificity trade-offs of using models of ...
17
votes
2answers
3k views

Can somebody explain to me NUTS in english?

My understanding of the algorithm is the following: No U-Turn Sampler (NUTS) is a Hamiltonian Monte Carlo Method. This means that it is not a Markov Chain method and thus, this algorithm avoids the ...
2
votes
1answer
139 views

Grid search methods for posterior distribution approximation

I'm reading the book "Bayesian Analysis with Python" and the author provides some python code designed to show the grid search method of obtaining an approximate posterior distribution for the classic ...
0
votes
1answer
31 views

What is the definition of a aperiodic Markov chain?

I understand the definition of a state being aperiodic or periodic with period d. But what does it mean for a chain to be aperiodic / periodic with period d? Thanks.
4
votes
2answers
2k views

MDP Value Iteration choosing gamma

What are the tradeoffs of choosing larger/smaller gamma when performing Value Iteration for MDPs? Will different values of gamma result in different policies?
0
votes
0answers
8 views

Quantiles of the Q values of an MDP

Cross-posted from Math StackExchange: Consider an MDP with $n$ states, $k$ actions, and discount factor $\gamma \in [0,1)$. We are uncertain of its reward function $R \in \mathbb{R}^{n \times k}$ and ...
5
votes
1answer
2k views

How to estimate the infinitesimal generator of a Markov chain?

I am working on an 8 state (8th state absorbing) multi-state markov model, and what I am having difficulty understanding is, how sensitive are the results to the initial qmatrix and what is the best ...
0
votes
0answers
89 views

How to bet on a binary event based on the markov transition matrix, state probabilities and the odds

There is a coupon full of football matches for a given day from a bookkeeper. I have scrapped another website and i have aquired continuous history of a particular match between ...
1
vote
2answers
788 views

Markov Chain and Removal Effect [closed]

Based on this article I'm trying to use within R the Channel Attribution package to leverage on the Markov Chain in order to attribute conversion between several marketing channel. On one point the ...
0
votes
1answer
145 views

Markov chain joint/conditional probability properties

I've been searching around for a bit and I can't find out if these kinds of operations are allowed as it's kind of exploiting the Markov property. Consider a Markov chain belonging to a the state ...
0
votes
0answers
10 views

Markov Chain Attribution Model: Calendar periods VS Sliding Window?

I'm trying to utilize Markov Chains in order to analyze and attribute online conversions of b2b users browsing on a company web. The key question mark I'm facing is on which period length to apply the ...
0
votes
1answer
24 views

Proving a Markov Chain (Random Walk) is Time-Homogeneous

Let $Y_0, Y_1,Y_2,... $ be a sequence of independent and identically distributed random variables. Then we define $X_n = \displaystyle\sum_{j=0}^{n} Y_j $ , $n=0,1,2,...$. This is a Random Walk ...
1
vote
1answer
29 views

Non-conditional probability to be in a given state in a markov chain [closed]

I am asking your help for a question I couldn't find the answer on the internet. I am considering a markov chain with 3 states. The transition probability matrix is pij, i, j from 1 to 3. pij is the ...
0
votes
0answers
10 views

HMM - Deal with Baum-Welch emission never observed

If I train a HMM with a given sequence of observations among n possible emissions, how do I deal with an emission that is never observed? For example, if in a 100 long observation sequence the ...
0
votes
1answer
905 views

Deriving Transition Matrix of the Embedded Markov Chain given the generator matrix?

Full Problem: A continuous-time Markov chain has generator matrix $$Q= \begin{pmatrix} -1 & 1 & 0 \\ 1 & -2 & 1 \\ 2 & 2 & -4 \\ \end{pmatrix} $$ (i) ...
1
vote
1answer
225 views

Randomly generating transition probabilities for Markov chains

I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ...
1
vote
1answer
46 views

Measures of interdependence of events of different types

A mouse does four things: eats (E), drinks (D), defecates (S), and urinates (U). A time chart records the occurrence of each event along with the time in minutes (after the commencement of measurement)...
0
votes
0answers
20 views

Why is this Markov Equivalence true?

I have the following markov relation $X_1 \leftarrow X \rightarrow X_2$, which leads to $X_1 \rightarrow X \rightarrow X_2$, but how does this lead to $X \rightarrow (X_1, X_2) \rightarrow X_2$ Can ...
0
votes
0answers
11 views

Geometric Brownian Motion with two-state diffusion/volatility

let's assume a GBM process S(t) with dynamics: dS(t) = a S(t) dt + b S(t) dB(t) where B(t) is a Brownian motion, a and b are constants, and S(0)>0. For any time s>t, we have that E_t[S(s)^k] = S(t)...
0
votes
0answers
12 views

Steady state distributions and stationary proabilities

Whats the difference between steady state distributions, long run probabilities and stationary distributions?
0
votes
0answers
39 views

Markov chain: expected number of visits to a state

Let $X_0$, $X_1$, $X_2$, ... be a Markov chain with state space 0, 1, 2, 3 and transition probabilities $$ \begin{matrix} 1/2 & 1/2 & 0 & 0\\ 1/3 & 1/3 & 1/3 & 0\\ 0 &...
1
vote
0answers
15 views

Number of observations required to approximate discrete random variable

I am trying to understand if, given a discrete random variable, there exists a formalised approach for determining the number of observations required to "well approximate" said variables true ...
0
votes
1answer
16 views

probability of getting to state a before state b starting from state a

If $X_n$ is a Markov Chain. P(0,0) = 0.5, P(0,1) = 0.5. For all states x > 0, P(x, x) = 0.5, P(x, x+1) = P(x, x-1) = 0.25. My goal is to find $P_0$ ($T_0$ < $T_5$), which is the probability of ...
0
votes
1answer
47 views

Machine Learning alternative for hashing [closed]

Is there a Machine Learning technique that can used to detect the slightest change in data? I know this can be done using a hash but I was just wondering if there is any machine learning technique out ...
0
votes
1answer
57 views

Markov Chain: transitioning to multiple states at the same time

I'm trying to calculate Customer Lifetime Value using Markov Chains. I'm following the paper by Pfeirer and Carraway. The paper evaluates CLV over a finite time horizon ...
3
votes
2answers
236 views

Markov process with future knowledge

Let's say we have a discrete-time Markov chain with a transition matrix $P$. If we know the initial state $x_0$, we can predict the probabilities of future states by iterating the chain as $x_{t+1}$ = ...
0
votes
1answer
48 views

Energy function of Restricted Boltzmann Machine (RBM)

The energy function for RBM (Restricted Boltzmann Machine) is defined as $$ E(v,h) = -\sum_{i,j} w_{ij} \, v_i \, h_j -\sum_i a_i \, v_i - \sum_i b_i \, h_i $$ with the joint distribution $$ \tag{1} p(...
1
vote
1answer
24 views

Birth Death process

An office has two employees that process incoming orders. these two are always busy and they process the orders at the rate of 100/day for each person. However they are smokers. On an average they ...
2
votes
1answer
330 views

Queuing Theory: Customers arrive at a fast-food restaurant at the rate of 120 customers per hour

During lunch hour, customers arrive at a fast-food restaurant at the rate of 120 customers per hour. The restaurant has one line, with three workers taking food orders at independent service stations. ...
0
votes
0answers
83 views

Variance reduction of an estimator arising from the marginal destribution of a Metropolis-Hastings chain

Let $(E,\mathcal E,\lambda)$ and $(E',\mathcal E',\lambda')$ be measure spaces $f\in L^2(\lambda)$ $I$ be a finite nonempty set $\varphi_i:E'\to E$ be bijective $(\mathcal E',\mathcal E)$-measurable ...
2
votes
0answers
19 views

Model or State Uncertainty in Queueing Model due to uncertain arrival rate

$\textbf{Introduction}$ I am currently modelling a scenario where two queues need to be served by a single server in a non preemptive discipline. I am quite sorted on generating the optimal policy ...
2
votes
1answer
62 views

How can we apply the rule of stationary distribution to the continuous case of Markov chain?

If the Markov chain converged then $$\pi = Q* \pi$$where $ \pi$ is the posterior distribution and $Q$ is the transition distribution(it's a matrix in the discrete case). I tried to apply that on the ...
3
votes
1answer
41 views

Can we calculate the theoretical stationary distribution from a continuous Markov chain?

I have the transition distribution $p(X_{t+1}|X_t=x_t) = \text{N}(\phi x_t,1)$ where $−1<\phi<1$. Can we calculate the stationary distribution and its mean and variance? I know I can do that ...
2
votes
1answer
44 views

Hidden-Markov Model for Markov-Chain with Sequential Bernoulli State Sampling

Consider a finite discrete-time Markov chain whose state is sampled at the times determined by the outcome of a Bernoulli process. That is, in each time period I flip a biased coin. If it comes up as "...
1
vote
1answer
48 views

Is the transition kernel of a Metropolis-Hastings chain of the form $P(x,A)=\varrho(x)\tilde P(x,A)+(1-\varrho(x))1_A(x)$?

After equation (1) at page 3 of this paper it is claimed that the transition kernel of a Markov chain generated by the Metropolis-Hastings algorithm is of the form $$P(x,A)=\varrho(x)\tilde P(x,A)+(1-\...
2
votes
1answer
504 views

Planning VS Reinforcement Learning for Large State Spaces

Does knowing everything about your environment yield any major shortcuts to finding the optimal policy, in a Markov Decision Process with a very large (finite) number of states? Mere planning clearly ...
2
votes
2answers
82 views

For a Markov Chain is $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$?

I am new to Markov Chains and using this concept in statistics. For a Markov Chain, may I say that $𝑃(𝑞_𝑡|𝑞_{𝑡+1},…,𝑞_𝑇)$ equals to $𝑃(𝑞_𝑡| 𝑞_{𝑡+1})$? If yes, how can I prove that?
1
vote
1answer
924 views

Measure the arrival rate or inter arrival time for a queueing model

I am analyzing the occurrence of emergencies in a given area and the application of queueing theory to determine the resources an emergency service should have ready in order to answer to emergency ...
0
votes
2answers
79 views

Suggest a model for this dataset

I have a time series data set (the Old Faithful geyser data available here: http://www.gatsby.ucl.ac.uk/teaching/courses/ml1-2012/geyser.txt). Plotting the eruption duration on the x axis and the ...
1
vote
2answers
53 views

Could a non-probabilistic state be a random variable?

This post gives a definition "of A stochastic process in discrete time" A stochastic process in discrete time n ∈ $N$ = {0, 1, 2, . . .} is a sequence of random variables (rvs) $X_0, X_1, X_2$, . . ...
2
votes
1answer
366 views

How is the Fermiac machine (Monte Carlo trolley) working?

There is a cool website showing the Markov chain with a machine. But nobody is explaining how it's working or showing a video of it's functioning. This is explaining the Markov chain monte carlo in ...
-1
votes
1answer
40 views

What is the n of Markov chain exactly equal to?

Section 7.2 of the book "transition probability graph" coming from the book "Introduction to Probability, 2nd Edition by Dimitri P. Bertsekas and John N. Tsitsiklis" gives some explanation of the ...
3
votes
0answers
568 views

How to estimate Markov chain transition probabilities with partially observed data?

Suppose that we have a time-homogeneous discrete-time Markov chain $(X_n)$. We want to estimate the transition probabilities $p_{ij} = \mathbb{P}[X_{n+1} = j \mid X_n = i]$. In the case when we have ...