A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms (1)

0
votes
0answers
14 views

Calculate state change probability

I am having telco order management data and need to calculate the probability of each order going through different stages. data is like this: Order No; product_type; time spent in step1; time ...
0
votes
0answers
15 views

Applying Markov Decision Processes to the Selling House Problem with waiting times

I'd like to apply the Markov Decision Process theory to this problem. We have a house to sell. Each day an offer of $X_n$ comes for the house. Each offer costs an amount $c$ to observe. You may think ...
1
vote
0answers
41 views

Which machine learning technique is appropriate for my problem?

I'm new in machine learning topics and I've problem in modeling my environment which has multi parameters with different value ranges and a few actions to perform when value of each parameter is not ...
3
votes
0answers
19 views

Testing Markov Property of transition matrix sequence with markovchain package in R

Related: Check memoryless property of a Markov chain I try to expand the verifyMarkovProperty function of the markovchain package to assess the Markov Property of transition matrices by assigning ...
1
vote
1answer
25 views

Notation in Markov Chain and MCMC literature

In the MC and MCMC literature one commonly finds statements of the following form (see e.g. Roberts & Rosenthal, 2004): $$ \int_{x \in \mathcal X} \pi(dx)P(x, dy) = \pi(dy). $$ What is the ...
0
votes
0answers
16 views

Markov chain with quota based 'immigration'

I am trying to simulate a KPI for my work, and while markov chains were covered briefly in the courses I attended for stats, I've never actually needed to use one until now. The difficulty I have is ...
2
votes
0answers
38 views

Markov Chain - Blackjack win problem [closed]

Got stuck with a problem from my text book. Need help please.. David is in Las Vegas, impressed by the Grandeur of the city, David decides try his luck at “Blackjack” in one of the casinos. David is ...
1
vote
1answer
54 views

Explain Backward algorithm for Hidden Markov Model

I have implemented Viterbi and Forward algorithm, alas strangely I can't understand how does Backward algorithm work. Intuitively I feel like I need to do the same thing as in Forward only backwards, ...
0
votes
0answers
16 views

Skeleton of a continuous Markov Chain

I have a continuous Markov Chain with transition matrix with initial state $X_0=1$ and state space $I=1,2,3,4,5$ $$P= \begin{bmatrix} -3 & 1 & 0 & 1 & 1\\ 0 & -1 & 0 & 0 ...
0
votes
0answers
9 views

Setting up transition matrix for balls in an urn

Here is the question I'm trying to answer. I don't need a solution for this, but I had a question about how I would approach the problem. An urn contains two red and two green balls. The balls are ...
5
votes
1answer
65 views

Markov chain,transition matrix and jordan form

If i have a transition matrix $$P= \begin{bmatrix} \frac12 & \frac14 & \frac14 \\ 0 & \frac12 & \frac12 \\ 0 & 0 & 1 \end{bmatrix}$$ i know that it's not diagonalizable,so if ...
0
votes
1answer
23 views

Markov Chain and classes property

I have a transition matrix P,i don't understand how my teacher explains properties of the states without any calculation. In example : $$P= \begin{array}{ccc} \frac12 & \frac14 & \frac14 \\ ...
1
vote
0answers
17 views

How to programmatically calculate decomposability index of continuous Markov Chain

I'm looking a way to calculate programmatically decomposability index of continuous Markov Chain. Could anyone post an example here or guide how to do it or give some code to show how it could be ...
0
votes
0answers
16 views

Markov process with only $n$ most significant transition probabilities known

Suppose I want to simulate a Markov process with a discrete state space. Normally, I need to have all the transition probabilities known. However, in my situation, I can only measure the top $n$ most ...
1
vote
1answer
32 views

How the Markov Chain Monte Carlo ensure the stationary distribution converge to the target distribution?

I am reading about the MCMC but now I got a lot of questions. Firstly, It says we could construct a markov chain which satisfy the detailed balance:...
0
votes
0answers
17 views

how to write down dynamical state space models with deterministic variables in PyMC?

is it possible to write down this simple dynamical system in pymc? $R_0 \sim Normal(\mu_r, \sigma_r)$ $Z_0 \sim Normal(\mu_z, \sigma_z)$ $R_t \sim Normal(R_{t-1}, \sigma_r)$ $Z_t = Z_{t-1} + ...
1
vote
0answers
15 views

using markov matrices for projections

Top of the morning folks .. Business context i am doing an experiment with an index time series (NSE, India's national stock exchange ..NIFTY to be precise) and i am using markov matrices to ...
1
vote
0answers
41 views

How can we say that $B_n$ is a Markov process (or something)?

From Probability with Martingales: I chose $\mathscr F_n = \sigma(B_1, B_2, ..., B_n)$. My argument assumes that $E[M_n | \mathscr F_{n-1}] = E[M_n | B_{n-1}]$. I was able to show that ...
0
votes
0answers
35 views

how to do MCMC simulation

I've been diving through the internet, trying to get the best possible understand on how to do this type of simulation, most of the information is theory or MCMC in simple english, but its been hard ...
0
votes
0answers
22 views

Stochastic Gradient on the Simplex

I have a probability density function defined over $x_1,...,x_K$ with a simplex constraint $\sum_{k=1}^K x_k = 1$. I'm trying to perform stochastic gradient descent on this density. I know I can keep ...
0
votes
1answer
24 views

Likelihood that a given outcome was generated by a Markov model

I am new to the concept of Markov Models and Markov Chain Monte Carlo simulations. I would like to take a piece of data and determine the likelihood that it was generated by a known MCMC ...
3
votes
2answers
96 views

Understanding Monte Carlo sampling

In rejection sampling or Markov chain Monte Carlo methods, we usually have a target distribution $p(x)$ whose form makes it difficult or impossible to draw samples directly, but we can evaluate $p(x)$ ...
0
votes
1answer
31 views

Meaning of 'compute' in the following statement

I am reading the following statement from a paper on the algorithms of inverse reinforcement learning: 'Using the RL Algorithm, compute the optimal policy $\pi$ for the MDP using the rewards $R = ...
1
vote
1answer
45 views

Markov Cluster Algorithm transition matrix

I am reading the notes on Markov Cluster Algorithm by Kathy Macropol (http://www.cs.ucsb.edu/~xyan/classes/CS595D-2009winter/MCL_Presentation2.pdf) On slide 14/46 the author talks about inflation and ...
0
votes
0answers
18 views

Additive Markov Chain in R

I have computed a customer lifecycle model for a university using the markovchain package, investigating transitional probabilities between courses. I would like to understand the impact that the ...
0
votes
2answers
53 views

In long run,Proportion of time in each state?

Consider a homogeneous Markov Chain with transition probability matrix P= Row 1=[0.5 0.4 0.1] Row 2=[0.3 0.4 0.3] Row 3=[0.2 0.3 0.5] In the long run what is the proportion of time is process in ...
3
votes
3answers
48 views

How do you see a Markov chain is irreducible?

I have some trouble understanding the Markov chain property irreducible. Irreducible is said to mean that the stochastic process can "go from any state to any state". But what defines whether it can ...
0
votes
0answers
21 views

Connection between chain rule and Markov property?

What's the connection between probability chain rule $$P(X)=P(X_L | X_{L-1},...,X_1)P(X_{L-1}|X_{L-2},...,X_1)P(X_1)$$ and the "key property of a (1st order) Markov chain": $$P(X)=P(X_L | ...
4
votes
0answers
53 views

How can I determine the expected value / risk of ruin of game where probability changes dependent on state?

Consider the following game: You are given a random number generator which you can use to play a game. The object of the game is to reach the final tier after which you collect prize tokens. In ...
3
votes
1answer
46 views

Measuring dependency of subsequent points from Markov chain

The question is about stimulating different type of species (coded 1-10) based on given species frequencies, and other parameters (eg. mean of normally distributed mass and ratio) using gibbs ...
1
vote
0answers
32 views

Is this application of strong Markov Property correct?

Let $X_n$ be a DTMC, with transition matrix P and state-space I. Let $Y_m=X_{T_m}$ for $m \in \mathbb{N}$. Define $T_0=\inf\{n\geq0:X_n\in J\subset I\}$ and $T_{m+1}=\inf\{n> T_{m}:X_n\in J\subset ...
0
votes
0answers
14 views

Compute partition function

I'm given a distribution on 3 discrete variables $x, y, z$ which is defined as $$p(x, y, z) = \frac{1}{Z} \psi(x, y, z) = \frac{1}{Z} \phi_1(x, y) \phi_2(y, z)$$ where $x, y$ can take up value among ...
2
votes
1answer
31 views

Probabilities in markov graphs

I have a first order markov graph which looks like Now I was told if we make B an absorbing state(NULL) the graph simplifies to I was also told that the conversion probability is 1 in the first ...
1
vote
2answers
53 views

Help me understand this: $ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \ $?

How do I read this equation (especially the left side) in terms of a Continuous Markov Process model? $$ Pr(T < t \mid y) = \int_{0}^{\infty }\int_{\Omega }k(x)p(t, x\mid y)\,dx\,dt \\ $$ Where $ ...
0
votes
0answers
29 views

Will n-th order Markov Chain have a better prediction than first order chain?

I am a student learning Stochastic Process right now. The Markov process was defined on a history independence hypothesis, however with some situations, the history-dependent data with Markovian could ...
1
vote
0answers
28 views

Stochastic process: Number of states visited in $n < \infty$

I am from Spain and I don't know the exact English term, so I will try to explain the definition. In a discrete stochastic process, let: P be the transition matrix with all is states being ...
0
votes
0answers
11 views

Comparing two or more transition probability matrices (msm)

Is there any statistics to compare between two (or more) transition probability matrices produced from a Markov multi-state model (the msm package in R)? I am analysing transitions in 3 different ...
1
vote
2answers
92 views

Algorithm for `rmarkovchain` in R

What method or algorithm is used in the function rmarkovchain from the R-package markovchain to generate samples and how does it work? Edit: I was interested in ...
2
votes
1answer
64 views

Creating a Markov Decision Process

So I have an idea for an MDP for my Machine Learning class, but I'm just wanting to make sure I take the right approach for setting it up. The Key Underlying question here is How do I approach a ...
1
vote
0answers
21 views

Likelihood over a arbitrary set of hypotheses

I have the following issue. I have finite, large, enumerated set $H$ of hypotheses that maps a time series to an integer, let's say days to integers i.e. $H \subseteq (\mathrm{Day} \to \mathbb{N})$. ...
0
votes
0answers
18 views

Reduce a transition matrix when using Markov chain to describe a web site activity

I have log from a website describing page by page visits. So lots of data. I would like to make a transition matrix with states each page of the site. So markov chains with continuous-time and ...
1
vote
0answers
21 views

Markov Chains: regularity and detailed balance [duplicate]

A Markov Chain with state transition matrix $A$ satisfies detailed balance with respect to distribution $\pi$ if for all $i,j$, we have $$\pi_iA_{i,j} = \pi_jA_{j,i}.$$ In such a case, we have ...
3
votes
0answers
78 views

Is there a measure of how well a Markov chain allows movement between states?

Define $$ A = \left( \begin{matrix} .5 & .5 \\ .5 & .5 \end{matrix} \right),\; \; B = \left( \begin{matrix} .99 & .01 \\ .01 & .99 \end{matrix} \right), \; \; C = \left( ...
1
vote
1answer
43 views

When is Markov chain a generator for iid sequences

Definition: Markov chain as : A stochastic process $X_1, X_2, \ldots, $ is a Markov process ( or Markov chain) is for any discrete time index $n = 1,2,\ldots,$ , $ Pr(X_{n+1} = x_{n+1}| X_n = x_n, ...
1
vote
1answer
31 views

State space for Markov Decision Processes

I'm currently trying to formulate a MDP for a Reinforcement Learning (RL) task. Having read a variety of papers where RL has been applied I've been left somewhat confused as to what can be a ...
0
votes
0answers
24 views

Portfolio optimization with user defined mean and variances

I found some functions for Markowitz mean variance portfolio optimization in R such as portfolio.optim in tseries package. ...
0
votes
1answer
14 views

Property of an irreducible Markov Chain

How can we prove that if a Markov Chain is irreducible (does not contain any closed set), then every state can be reached from every other state in the chain ?
2
votes
1answer
36 views

Equivalent definition of persistence of state in Markov Chain

Let $E_j$ be a particular state in a sequence of finite states that qualify to follow the Markov Chain Property. If $E_j$ is persistent then by definition, \begin{equation} f_{jj}=\sum_{n=1}^{\infty} ...
3
votes
1answer
51 views

Are the mean of samples taken from Metropolis-Hasting MCMC normally distributed?

I've come across the following theorem while studying MCMC. It seems to suggest that the sample mean taken from the MCMC – the posterior marginal expectation – should be normally distributed, using ...
0
votes
0answers
25 views

What type of model am I looking for?

I will explain by way of an example. Say I have data on 2 groups of people (boys and girls). I would like to learn 1 model for girls, 1 model for boys. The variables to consider are things like what ...