A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms

0
votes
0answers
3 views

What are good tutorial on Weighted Finite Automata?

I would especially appreciate papers, books or tutorials with source code already available. Currently I'm reading "Spectral Learning Techniques for Weighted Automata, Transducers, and Grammars" by ...
1
vote
1answer
13 views

Estimating Markov Switching Probit

I attempt to fit the following probit model to a time series where we observe the binary variable $R_{t}$ and another variable $X_{t}$, a latent unobserved variable $y^{*}_{t}$ and a state variable ...
0
votes
0answers
13 views

Distribution of the duration of a markov-process in a specified state during a specified time

I have a continuous time markov chain with two states $A$ and $B$. The transition rate $A\rightarrow B$ is $\lambda$ and $B\rightarrow A$ is $\mu$. Imagine that $P(X{t_0}=A)=1$ (the process starts in ...
0
votes
0answers
15 views

disease progression R markov chains

hello i have a dataframe, some of the columns include: remission,height,weight,time from diagnosis, age ethnicity, age, patient id remission is 1 or 0 just to be clear i want to fit an appropriate ...
0
votes
0answers
23 views

Subsetting dataframe conditions on factor (binary) column (vector in R)

I have a sequence of 1/0's indicating if patient is in remission or not. Assume the records of remission or not were taken at discrete times. How can I check the Markov property for each patient, ...
1
vote
1answer
20 views

r markov chain markov property on binary variable, discrete time

i have a sequence of 1/0's indicating if patient is in remission or not, assume the records of remission or not were taken at discrete times, how can i check the markov property for each patient, ...
1
vote
0answers
40 views

Fit and evaluate a second order transition matrix (Markov Process) in R?

I already built 1 first order discrete state Markov Chain model. It was built with R using the function 'markovchainFit()' in ...
0
votes
0answers
14 views

N-gram learning vs stochastic learning

I'm interested in comparing the differences in learning in n-grams and gradient-based learning (in my case with neural networks), particularly in the context of language modelling with the two classes ...
0
votes
0answers
20 views

Markov chains hitting times

I'm having trouble understanding what hitting times are in markov chain processes and how they are calculated. An example follows: A Markov process on E = {1, 2, 3} has the following generator matrix ...
0
votes
1answer
35 views

Markov Process-Variance of time until jump

A Markov process on E = {1, 2} is constructed according to holding time parameters λ1 = 2 and λ2 = 4; the defining Markov chain has transition probabilities p11 = p12 = 0.5 and p21 = 1. How do I ...
0
votes
1answer
11 views

Finding One Step Transition Matrix in Gambling?

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
-1
votes
1answer
23 views

Does the order of variables in a Markov Regime Switching model matter?

since Ive received feedback that my previous question was not well-recieved Ill just have to give it another shot. I am estimating Markov Regime Switching Models, and I am getting different results ...
3
votes
0answers
47 views

simulating birth death process with random numbers from negative binomial

I am trying to generate random deviates for the population size at time $t$ for a birth-death process with constant birth and death rates per individual and initial size $N_0 \gt 0$. For the simple ...
0
votes
0answers
12 views

Real-life examples of Markov Decision Processes

I've been watching a lot of tutorial videos and they are look the same. This one for example: https://www.youtube.com/watch?v=ip4iSMRW5X4 They explain states, actions and probabilities which are ...
0
votes
1answer
12 views

Birth & Death process - Combining Transition rates

I think I'm missing a fundamental step in regards to how to combine two exponential distributions in the context of this problem. If we have a birth and death process where birth rate ~ ...
0
votes
1answer
50 views

writing down markov chain transition matrix

Question: An experimental animal can stay in room-A until 1 minute,and it can stay in room-B until 2 minutes. There exist deadly gases in room-C. One room among these three rooms is being randomly ...
0
votes
0answers
14 views

Generating markovian paths

given a 4 by 4 transition matrix P, I want to simulate m Markovian paths of length n I created this function can anyone tell me if this is is correct ? ...
0
votes
0answers
24 views

Markov Process w/ a non-stochastic matrix?

I've come across a problem which at first appeared to be a markov process however the transition matrix of the graph is non-stochastic. That is, the probabilities among edges leaving a node do not sum ...
0
votes
0answers
35 views

Decomposing the non-deterministic transition functions in non-Markov decision processes into several deterministic transition functions

Problems in reinforcement learning are commonly modeled as Markov decision processes (MDPs). One essential part of MDPs is the transition function $T: S \times A \times S \rightarrow [0, 1] \in ...
0
votes
0answers
30 views

Best textbooks on Non-Homogeneous Stochastic Processes?

just wanted to know which are in your opinion some of the best available books on theory and applications of NH Poisson Stochastic Processes, and Non-Poisson processes out there. I've studied Parzen ...
0
votes
0answers
24 views

How to fit a stochastic matrix to given data.?

Given a data sequence of noisy observations of a 3-state Markov chain $X$ -- $y_1$,$y_2$,...$y_n$, with two transition matrices $A_1$ and $A_2$ corresponding to different regions (**) in the (unit) ...
0
votes
1answer
18 views

What statistic to use to measure effectiveness of treatment on fluctuating process

I have a process $R$ that normally does something like a random walk between 0 and 1. I have a set of treatments. I believe that some of the treatments will bias the process $R$ in such a way that, ...
0
votes
0answers
17 views

Fit a transition matrix

Probably I am asking something very simple, but I am unable to figure out the answer from previous posts. I have a set of data with the age (in years) an the condition (in a class form 1 - good to 5 - ...
3
votes
1answer
94 views

Confusion in Gibbs sampling

I am self-studying Gibbs sampling from a book. The book introduces metropolis hastings algortihm to generate representative values from a posterior distribution. So we know $p(D | \theta) p(\theta)$ ...
0
votes
0answers
25 views

Comparing Two Markov Processe with Known Transition Rate Matrices

I have a Markov process (Continuous time) with the transition matrix "A". I also have an estimation of the transition matrix named "B". How can I understand whether B is a good estimation for A. I ...
0
votes
1answer
25 views

MDP Value Iteration choosing gamma

What are the tradeoffs of choosing larger/smaller gamma when performing Value Iteration for MDPs? Will different values of gamma result in different policies?
2
votes
1answer
106 views

Kalman filter with control inputs in python?

i am trying to fit a simple kalman filter with input controls (in this case step input) in python. i am using filterpy (http://filterpy.readthedocs.org/). my code is: ...
4
votes
1answer
96 views

Gibbs Sampler transition kernel

Let $\pi$ be the target distribution on $(\mathbb{R}^d,\mathcal{B}(\mathbb{R^d}))$ which is absolutely continuously wrt to the $d$-dimensional Lebesgue measure, i.e : $\pi$ admits a density ...
5
votes
1answer
184 views

Gibbs Sampler contradiction proof

I want to prove that the systematic scan Gibbs sampler yields an aperiodic chain $X$ on a general state space. Let $\pi$ be the stationary distribution for the resulting chain. Suppose to get a ...
2
votes
1answer
64 views

Developing Markov Transition Matrix

I would like to build a transition matrix based on some tabular data given that: I have about 50,000 historical data points Data is organized in a way such as ...
0
votes
0answers
28 views

How long does it take two identical hidden Markov models run on same observations to forget their initial distributions (if ever)?

Let $H_1$ and $H_2$ be two instances of a finite Hidden Markov Model (HMM) $H$. That is, $H_1$ and $H_2$ have identical state spaces $Q$ as well as identical transition $A$ and emission probabilities ...
1
vote
0answers
51 views

Generating function from the infinitesimal generator of a continuous-time markov chain?

Summary: The basic goal is to find the time evolution of the probability generating function (or the moment generating function or the characteristic function if you prefer) for a continuous-time ...
2
votes
0answers
67 views

how to determine if two dice are fair using pymc and roll data

My scenario is that I have two six-sided dice (D1 and D2), either of which may be fair or loaded (biased). I have samples of combined roll data (i.e. D1 + D2). I would like to view the posterior ...
0
votes
0answers
17 views

Connection between discrete VAR(1) model and simple discrete Markov Chain

I have studied both Markov chains and Vector Autoregressive Models, and I am interested in the connections between the following models: Markov Chain: $$X_{t+1}=T*X_{t}$$ Where X is a vector ...
0
votes
0answers
35 views

Repeated utility values in Value Iteration (Markov Decision Process)

I am trying to implement the value iteration algorithm of the Markov Decision Process using python. I have one implementation. But, this is giving me many repeated values for the utilities. My ...
0
votes
0answers
18 views

Problem related to variance of first passage matrix of a absorbing Markov chain

Consider the below computations taken from Kemeny/Snell Finite Markov Chains. Here $N=(I-Q)^{-1}$ calculated from some absorbing MC. $N_2$ is the variance matrix of $N$ and $N_{sq}$ is taken by ...
0
votes
1answer
66 views

How to determine second order Markov chain by transition probability

The following Eqation is for the first order, how can I write it for second order? where |P is the transition probability.
1
vote
1answer
39 views

Interpreting the mean first passage matrix of a Markov chain

Consider the following first passage matrix: I just want to know whether one can give a good interpretation to this matrix. All I know to say is that it takes this long to go from this state to ...
2
votes
1answer
75 views

Can Hidden Markov Models be used to predict next observation?

I am reading up on Hidden Markov Models (HMMs) for my research and would like to know if it is applicable to the problem I wish to tackle. My problem is to detect/estimate the next value of a ...
0
votes
0answers
19 views

Learning pairwise influence of binary variables (networks)

I have data of 110 observations of 750 binary variables each sampled during 40 time periods. What I would like to figure out is, for each pair of variables, what probabilistic influence at time t do ...
0
votes
0answers
13 views

Learning from sparse label products

Consider the following binary classification problem $f(\mathbf{X})\rightarrow\mathbf{Y}$ where: $\mathbf{X}$ = Feature matrix $\mathbf{Y}$ = Product of several label (binary) vectors, i.e. ...
1
vote
0answers
91 views

R - system is computationally singular - dealing with small numbers

I'm working with a ~200x200 Markovian transition matrix of non-zero probabilities. Forcibly, these probabilities are, for the large part, going to be very small. I am trying to find the inverse of my ...
4
votes
0answers
47 views

Calculating the first time a particle hits a state

Let $(X_{n})$ be a Markov chain with state space $D=(a,b,c)$ and transition matrix $$P= \pmatrix{ 0.4 & 0.6 & 0 \\ 0.5 & 0 & 0.5 \\1 & 0 & 0 \\}$$ A) Find the lim$_{n-> ...
1
vote
1answer
43 views

Steady state Markov Chain

is it able to count the steady state of problem with recurrent subchain.. for example if there are A B C D things and they are all recurrent. do they have steady state?? and also.. how to count ...
1
vote
1answer
72 views

Fitting data to a Markov Chain

I am looking for the reference/toolbox/note on how to fit a finite discrete-time Markov Chain to given time series. Ideally, there shall also be criteria of whether the fit is good, and whether ...
2
votes
2answers
61 views

What distribution is this? Activated process

A particle randomly hops a discrete distance from one position to another. I have measured, for 200 hops, the time between each hop. Here is the histogram: What distribution is this? To look at, ...
0
votes
0answers
22 views

game scoring marginal probability estimation

I have a game scoring time series data, where X[i] shows the score of player 1 minus player 2 after the ith turn. The game is played in turns, meaning if player 1 plays at ith turn, player 2 plays at ...
1
vote
1answer
22 views

Embedding Markov Matrix

A stochastic matrix with states $S_1$, $S_2$, $S_3$, $S_4$ is given, now we would like to build up another stochastic matrix with finer states, meaning that the states $S_1$ will be considered as ...
2
votes
0answers
69 views

Probability that uniformly distributed points in a square region form a cluster

I have a known number of points N uniformly distributed in a square and I want to solve the expected number of clusters of points. I cluster is formed by a growing algorithm. Starting at a point p, ...
0
votes
0answers
37 views

Markov Chain to generate random Words

I would like to generate 1000 words (for example) using a Markov matrix of alphabets probabilities. It will be helpful if Matlab code provided. ...