A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms

0
votes
0answers
18 views

Markov Process w/ a non-stochastic matrix?

I've come across a problem which at first appeared to be a markov process however the transition matrix of the graph is non-stochastic. That is, the probabilities among edges leaving a node do not sum ...
0
votes
0answers
28 views

Decomposing the non-deterministic transition functions in non-Markov decision processes into several deterministic transition functions

Problems in reinforcement learning are commonly modeled as Markov decision processes (MDPs). One essential part of MDPs is the transition function $T: S \times A \times S \rightarrow [0, 1] \in ...
0
votes
0answers
27 views

Best textbooks on Non-Homogeneous Stochastic Processes?

just wanted to know which are in your opinion some of the best available books on theory and applications of NH Poisson Stochastic Processes, and Non-Poisson processes out there. I've studied Parzen ...
0
votes
0answers
21 views

How to fit a stochastic matrix to given data.?

Given a data sequence of noisy observations of a 3-state Markov chain $X$ -- $y_1$,$y_2$,...$y_n$, with two transition matrices $A_1$ and $A_2$ corresponding to different regions (**) in the (unit) ...
0
votes
1answer
17 views

What statistic to use to measure effectiveness of treatment on fluctuating process

I have a process $R$ that normally does something like a random walk between 0 and 1. I have a set of treatments. I believe that some of the treatments will bias the process $R$ in such a way that, ...
0
votes
0answers
12 views

Fit a transition matrix

Probably I am asking something very simple, but I am unable to figure out the answer from previous posts. I have a set of data with the age (in years) an the condition (in a class form 1 - good to 5 - ...
2
votes
1answer
86 views

Confusion in Gibbs sampling

I am self-studying Gibbs sampling from a book. The book introduces metropolis hastings algortihm to generate representative values from a posterior distribution. So we know $p(D | \theta) p(\theta)$ ...
0
votes
0answers
21 views

Comparing Two Markov Processe with Known Transition Rate Matrices

I have a Markov process (Continuous time) with the transition matrix "A". I also have an estimation of the transition matrix named "B". How can I understand whether B is a good estimation for A. I ...
0
votes
1answer
10 views

MDP Value Iteration choosing gamma

What are the tradeoffs of choosing larger/smaller gamma when performing Value Iteration for MDPs? Will different values of gamma result in different policies?
2
votes
1answer
79 views

Kalman filter with control inputs in python?

i am trying to fit a simple kalman filter with input controls (in this case step input) in python. i am using filterpy (http://filterpy.readthedocs.org/). my code is: ...
4
votes
1answer
82 views

Gibbs Sampler transition kernel

Let $\pi$ be the target distribution on $(\mathbb{R}^d,\mathcal{B}(\mathbb{R^d}))$ which is absolutely continuously wrt to the $d$-dimensional Lebesgue measure, i.e : $\pi$ admits a density ...
5
votes
1answer
180 views

Gibbs Sampler contradiction proof

I want to prove that the systematic scan Gibbs sampler yields an aperiodic chain $X$ on a general state space. Let $\pi$ be the stationary distribution for the resulting chain. Suppose to get a ...
2
votes
1answer
51 views

Developing Markov Transition Matrix

I would like to build a transition matrix based on some tabular data given that: I have about 50,000 historical data points Data is organized in a way such as ...
0
votes
0answers
27 views

How long does it take two identical hidden Markov models run on same observations to forget their initial distributions (if ever)?

Let $H_1$ and $H_2$ be two instances of a finite Hidden Markov Model (HMM) $H$. That is, $H_1$ and $H_2$ have identical state spaces $Q$ as well as identical transition $A$ and emission probabilities ...
1
vote
0answers
43 views

Generating function from the infinitesimal generator of a continuous-time markov chain?

Summary: The basic goal is to find the time evolution of the probability generating function (or the moment generating function or the characteristic function if you prefer) for a continuous-time ...
2
votes
0answers
60 views

how to determine if two dice are fair using pymc and roll data

My scenario is that I have two six-sided dice (D1 and D2), either of which may be fair or loaded (biased). I have samples of combined roll data (i.e. D1 + D2). I would like to view the posterior ...
0
votes
0answers
14 views

Connection between discrete VAR(1) model and simple discrete Markov Chain

I have studied both Markov chains and Vector Autoregressive Models, and I am interested in the connections between the following models: Markov Chain: $$X_{t+1}=T*X_{t}$$ Where X is a vector ...
0
votes
0answers
25 views

Repeated utility values in Value Iteration (Markov Decision Process)

I am trying to implement the value iteration algorithm of the Markov Decision Process using python. I have one implementation. But, this is giving me many repeated values for the utilities. My ...
0
votes
0answers
13 views

Problem related to variance of first passage matrix of a absorbing Markov chain

Consider the below computations taken from Kemeny/Snell Finite Markov Chains. Here $N=(I-Q)^{-1}$ calculated from some absorbing MC. $N_2$ is the variance matrix of $N$ and $N_{sq}$ is taken by ...
0
votes
1answer
40 views

How to determine second order Markov chain by transition probability

The following Eqation is for the first order, how can I write it for second order? where |P is the transition probability.
1
vote
1answer
37 views

Interpreting the mean first passage matrix of a Markov chain

Consider the following first passage matrix: I just want to know whether one can give a good interpretation to this matrix. All I know to say is that it takes this long to go from this state to ...
2
votes
1answer
66 views

Can Hidden Markov Models be used to predict next observation?

I am reading up on Hidden Markov Models (HMMs) for my research and would like to know if it is applicable to the problem I wish to tackle. My problem is to detect/estimate the next value of a ...
0
votes
0answers
19 views

Learning pairwise influence of binary variables (networks)

I have data of 110 observations of 750 binary variables each sampled during 40 time periods. What I would like to figure out is, for each pair of variables, what probabilistic influence at time t do ...
0
votes
0answers
13 views

Learning from sparse label products

Consider the following binary classification problem $f(\mathbf{X})\rightarrow\mathbf{Y}$ where: $\mathbf{X}$ = Feature matrix $\mathbf{Y}$ = Product of several label (binary) vectors, i.e. ...
1
vote
0answers
72 views

R - system is computationally singular - dealing with small numbers

I'm working with a ~200x200 Markovian transition matrix of non-zero probabilities. Forcibly, these probabilities are, for the large part, going to be very small. I am trying to find the inverse of my ...
4
votes
0answers
47 views

Calculating the first time a particle hits a state

Let $(X_{n})$ be a Markov chain with state space $D=(a,b,c)$ and transition matrix $$P= \pmatrix{ 0.4 & 0.6 & 0 \\ 0.5 & 0 & 0.5 \\1 & 0 & 0 \\}$$ A) Find the lim$_{n-> ...
1
vote
1answer
34 views

Steady state Markov Chain

is it able to count the steady state of problem with recurrent subchain.. for example if there are A B C D things and they are all recurrent. do they have steady state?? and also.. how to count ...
1
vote
1answer
54 views

Fitting data to a Markov Chain

I am looking for the reference/toolbox/note on how to fit a finite discrete-time Markov Chain to given time series. Ideally, there shall also be criteria of whether the fit is good, and whether ...
2
votes
2answers
61 views

What distribution is this? Activated process

A particle randomly hops a discrete distance from one position to another. I have measured, for 200 hops, the time between each hop. Here is the histogram: What distribution is this? To look at, ...
0
votes
0answers
20 views

game scoring marginal probability estimation

I have a game scoring time series data, where X[i] shows the score of player 1 minus player 2 after the ith turn. The game is played in turns, meaning if player 1 plays at ith turn, player 2 plays at ...
1
vote
1answer
22 views

Embedding Markov Matrix

A stochastic matrix with states $S_1$, $S_2$, $S_3$, $S_4$ is given, now we would like to build up another stochastic matrix with finer states, meaning that the states $S_1$ will be considered as ...
2
votes
0answers
63 views

Probability that uniformly distributed points in a square region form a cluster

I have a known number of points N uniformly distributed in a square and I want to solve the expected number of clusters of points. I cluster is formed by a growing algorithm. Starting at a point p, ...
0
votes
0answers
30 views

Markov Chain to generate random Words

I would like to generate 1000 words (for example) using a Markov matrix of alphabets probabilities. It will be helpful if Matlab code provided. ...
0
votes
0answers
21 views

Compute Transition Matrix [duplicate]

I am studying Markov Chain. I want to compute transition matrix. Is this the right way to do it ...
0
votes
0answers
28 views

Have a data set for 3 consecutive days. What are my options?

Let's say I have a set regarding the transportation methods(Eg: car, bus, train) used for three consecutive days b y $n$ number of people. For simplicity let us assume that everyone use only one type ...
0
votes
0answers
23 views

Modelling remaing time of a process

I have process which has different states. It looks something like this: In some cases the required tools for the assembly need to be fetched (same goes for the supplies for packaging). Typical ...
0
votes
1answer
99 views

Estimating Standard Errors for Markov Transition Probability with Multiple Observations (in R)

I was trying to estimate a Markov transition table from paired transition data, which look something like this: ...
-1
votes
1answer
39 views

Markov Chains : Can anything be said about what happens in between two transition?

In time homogeneous discrete Markov chains we take a set period for a single transition. In examples we see sometimes depending on the examples the transition period being a a month a week etc. I'm ...
0
votes
1answer
23 views

Modelling a probability distribution on different feature sets

I have a binary classification problem, and I use method A and method B to extract features, F1 and F2, for this problem from dataset X. Now, I train two models, y1 and y2, separately on the two ...
0
votes
0answers
13 views

sample generative model from a chain/tree

I have a tree of states and I would like to sample from this tree based on pure birth process; however, I don't know how exactly I can do this; so far I have done this; I simplified my problem; the ...
0
votes
0answers
37 views

Probabilities in a Markov Model

I am reading a paper on Markov Models and I am trying to figure out how to compute the probabilities for the $\alpha$-pass. I am given an $N\times N$ matrix $A$, that has the probabilities of ...
1
vote
1answer
47 views

MDP value iteration

In Markov decision processes, what is the guarantee that value iteration chooses the same policy action from a given state for every iteration? I am referring to the slides given by AWM at ...
0
votes
0answers
40 views

Maxent Markov models in R

Is there a package that implements Maxent Markov Models in R? (http://en.wikipedia.org/wiki/Maximum-entropy_Markov_model). I understand that package crf implemets conditional random fields which are ...
2
votes
1answer
27 views

Is it possible to determine the probability of NOT reaching a state in a FSA ?

Suppose I have a Finite State Automata with various states and probabilities of state transitions. Does mathematics exist to determine the probability of NOT reaching a state in the FSA given some n ...
1
vote
1answer
104 views

Transition rates in continuous time markov chain

A house has 2 rooms of similar sizes with identical air conditioners equipped with thermostats which turn on and off as needed to maintain the temperature in each room to a desired level of 22 ...
0
votes
0answers
27 views

Transition matrix in left-right hidden semi-Markov model

i'm developing a hidden semi-Markov model left-right . In a left-right model a sequence of $M$ states starts in state 1 and ends in state M, with no repetition of states. Since the model is ...
4
votes
0answers
92 views

How Should I Find Statistically Significant Differences Between Two Markov Models?

Suppose I have N Markov Models of M states representing the behaviour patterns of 2 different groups (note: fully observable models, no hidden states), and have stored each model as a matrix of ...
3
votes
0answers
36 views

Convergence time of a Markov chain

We know that a regular Markov chains converges to a unique matrix. The convergence time maybe finite or infinite. My interest is in the case where the convergence time is finite. How can we accurately ...
0
votes
0answers
53 views

How would you validate a random walk model?

I have used a random walk model and Gibbs sampling (more specifically RJAGS) in order to obtain posterior of the state given the observations. In this case the state is the true proportion of the ...
1
vote
0answers
53 views

Concavity of log likelihood for hidden markov models

Could you give me a good link where the concept of concavity of the log likelihood related to hidden markov model EM algorithm is clarified? Thank you in advance.