A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms (1)

0
votes
0answers
2 views

Reference help MCMC

I am studying on Markov chains, random walks, gibs chains and optimizations using methods such as bisection and Newton Raphson, but am finding that the references that I have a little complex, like to ...
0
votes
0answers
19 views

Binning by standard deviations

Quick question: I came across a fairly respected source on running Markov Chain Monte Carlo for bayesian statistics in ...
0
votes
0answers
14 views

Estimating transition probabilities from the sequences without repeated states

Is it possible to estimate transition probabilities of the continuous process if we observe only the first occurence of the state? For example, if the real transition sequence was 1 - 5 seconds; 2 - ...
2
votes
0answers
29 views

Markov models that with several active states

Are there any Markov-like models that can have several active states? So say if trying to determine (the chance) when the person will wake up based on two variables (weather and the time the person ...
0
votes
1answer
21 views

Can reinforcement learning be used as path guidance?

I have an online path guidance system that learns from a set of past experiences(trajectories) to provide a guidance for the user on how to cover the given space in the best way, adopting to whatever ...
0
votes
0answers
21 views

markov chain question [on hold]

There are two points which are A and B. The distance between A and B is 50meter. One person goes to A with probability 1/6, he goes to B with probability 3/6. And he goes nowhere with probability 2/6. ...
0
votes
0answers
23 views

anomaly detection with Markov chain

The paper uses a simple technique to detect intrusions in computer systems. I will briefly explain it and ask a question: The paper proposes a simple 1-order Markov chain modelling approach to detect ...
0
votes
1answer
35 views

markov chain - probability question

Transition matrix has been written like that; $$\mathcal P = \begin{bmatrix} 1/3 & 0 & 2/3 \\ 1/3 & 1/3 & 1/3 \\ 0 & 0 & 1 \end{bmatrix}$$ the initial vector is that ...
1
vote
0answers
12 views

How to estimate Markov chain transition probabilities with partially observed data?

Suppose that we have a time-homogeneous discrete-time Markov chain $(X_n)$. We want to estimate the transition probabilities $p_{ij} = \mathbb{P}[X_{n+1} = j \mid X_n = i]$. In the case when we have ...
0
votes
1answer
18 views

Does there exist an algorithm/package in R that tells if a matrix is aperiodic? [closed]

I am currently reading Golub and Jacksons paper "Naïve learning in social networks and the wisdom of crowds" published in the American Economic Journal: Microeconomics in 2010. On page 120 they say ...
1
vote
0answers
19 views

Measuring effectiveness of marketing through attribution analysis [on hold]

My data(dataframe in R) looks like this:The data is ordered by CustomerName and then TimeofEvent. ...
3
votes
0answers
31 views

Markov decision process in R for a song suggestion software?

We have a music player that has different playlists and automatically suggests songs from the current playlist I'm in. What I want the program to learn is, that if I skip the song, it should decrease ...
0
votes
0answers
5 views

Identifying sequences of behavioral interactions between multiple individuals

I'm wondering if anyone might have some novel insights as to the best way to analyze the following data. It's a problem I've been thinking about in the back of my mind for a while, so I thought that ...
2
votes
1answer
57 views

Markov chains with a stationary distribution but no limiting distribution

I am trying to intuitively reconcile the following statement, read from "Probability, Markov Chains, and Queues": ...
1
vote
0answers
14 views

First-order Discrete Markov Chain with time lag

I want to estimate the first-order transition matrix of a sequence in discrete time, e.g. $$ s = 1,0,1,0,1,1,0,1,0,0, \dots$$ but states are not evenly spaced in time. So that even if $s_{t=1} = 1$ ...
0
votes
0answers
6 views

Estimate conditional probability of connected variables

I want to model the following probability function $p(x_i|\mathcal{N}_{x_i})$, where $\mathcal{N}_{x_i}$ is the set of the variables $x_j$ conneced to $x_i$ given a specified undirected graph ...
0
votes
0answers
14 views

Degrees of Freedom for Inhomogeneous Markov Chains

Suppose I am given a time-inhomogeneous Markov chain. For simplicity, let's assume we have 5 (time) positions, and 4 states (A,B,C,D). How can I compute the degrees of freedom for a first order model ...
0
votes
1answer
38 views

What is the expected amount of time it takes to first observe a triple for multiple dice rolls? [duplicate]

Assuming we have a fair die, we toss a die multiple times. Also assuming that a triple is defined as when we have three rolls in a roll that result in the same number, and that the rolls are ...
1
vote
0answers
37 views

Analyzing output in MCMC

I am using emcee to do inference on some data. I am trying to fit my data to a line of equation $ y = mx + b $. ...
4
votes
1answer
41 views

Given two absorbing Markov chains, what is the probability that one will terminate before the other?

I have two different Markov chains, each with one absorbing state and a known starting position. I want to determine the probability that chain 1 will reach an absorbing state in fewer steps than ...
0
votes
0answers
20 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
0
votes
0answers
4 views

What are good tutorial on Weighted Finite Automata?

I would especially appreciate papers, books or tutorials with source code already available. Currently I'm reading "Spectral Learning Techniques for Weighted Automata, Transducers, and Grammars" by ...
1
vote
1answer
23 views

Estimating Markov Switching Probit

I attempt to fit the following probit model to a time series where we observe the binary variable $R_{t}$ and another variable $X_{t}$, a latent unobserved variable $y^{*}_{t}$ and a state variable ...
-1
votes
1answer
20 views

Proof of Markov Chain property

Suppose that $X_n$ is a Markov Chain.Then for $m,n \in N$ such that $m<n$ $Pr[X_n=j_n|X_m=j_m,X_{m-1}=j_{m-1},...=X_0=j_0]=Pr[X_n=j_n|X_m=j_m]$ When proving for n=3,m=1 case we have to show ...
1
vote
1answer
15 views

Conditional transition matrix (consistent estimates)

Is there a model/technique that is able to estimate transition matrix (which would be consistent, i.e. sums of their rows would be always 1) conditional on some continuous variable X? Let's say I ...
0
votes
0answers
17 views

Distribution of the duration of a markov-process in a specified state during a specified time

I have a continuous time markov chain with two states $A$ and $B$. The transition rate $A\rightarrow B$ is $\lambda$ and $B\rightarrow A$ is $\mu$. Imagine that $P(X{t_0}=A)=1$ (the process starts in ...
0
votes
0answers
15 views

disease progression R markov chains

hello i have a dataframe, some of the columns include: remission,height,weight,time from diagnosis, age ethnicity, age, patient id remission is 1 or 0 just to be clear i want to fit an appropriate ...
0
votes
0answers
33 views

Subsetting dataframe conditions on factor (binary) column (vector in R)

I have a sequence of 1/0's indicating if patient is in remission or not. Assume the records of remission or not were taken at discrete times. How can I check the Markov property for each patient, ...
1
vote
1answer
25 views

r markov chain markov property on binary variable, discrete time

i have a sequence of 1/0's indicating if patient is in remission or not, assume the records of remission or not were taken at discrete times, how can i check the markov property for each patient, ...
1
vote
0answers
67 views

Fit and evaluate a second order transition matrix (Markov Process) in R?

I already built 1 first order discrete state Markov Chain model. It was built with R using the function 'markovchainFit()' in ...
0
votes
0answers
17 views

N-gram learning vs stochastic learning

I'm interested in comparing the differences in learning in n-grams and gradient-based learning (in my case with neural networks), particularly in the context of language modelling with the two classes ...
0
votes
0answers
22 views

Markov chains hitting times

I'm having trouble understanding what hitting times are in markov chain processes and how they are calculated. An example follows: A Markov process on E = {1, 2, 3} has the following generator matrix ...
0
votes
1answer
40 views

Markov Process-Variance of time until jump

A Markov process on E = {1, 2} is constructed according to holding time parameters λ1 = 2 and λ2 = 4; the defining Markov chain has transition probabilities p11 = p12 = 0.5 and p21 = 1. How do I ...
0
votes
1answer
19 views

Finding One Step Transition Matrix in Gambling?

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
-1
votes
1answer
24 views

Does the order of variables in a Markov Regime Switching model matter?

since Ive received feedback that my previous question was not well-recieved Ill just have to give it another shot. I am estimating Markov Regime Switching Models, and I am getting different results ...
1
vote
0answers
26 views

How do you show that a Markov chain has not mixed?

I came across the claim that when doing Gibbs sampling, it is possible to show that a Markov chain has not yet mixed. How is this done?
2
votes
2answers
34 views

Distribution of p(x) in empirical model

I am having a hard time to exactly name what I am looking for (I am quite sure it already exists out there...) so I'll start with a concrete example: I have a population of discrete colours (red, ...
8
votes
1answer
90 views

Probability of a consecutive pair of values

Lets $X=(x_1, x_2,...x_{20})$ where $x_i\sim N(0,1)$ and $x_i, x_j$ are independent $\forall i\neq j$. What is the probability to obtain a sample $X$ where there are at least two consecutive values ...
3
votes
0answers
61 views

simulating birth death process with random numbers from negative binomial

I am trying to generate random deviates for the population size at time $t$ for a birth-death process with constant birth and death rates per individual and initial size $N_0 \gt 0$. For the simple ...
0
votes
0answers
18 views

Real-life examples of Markov Decision Processes

I've been watching a lot of tutorial videos and they are look the same. This one for example: https://www.youtube.com/watch?v=ip4iSMRW5X4 They explain states, actions and probabilities which are ...
0
votes
1answer
13 views

Birth & Death process - Combining Transition rates

I think I'm missing a fundamental step in regards to how to combine two exponential distributions in the context of this problem. If we have a birth and death process where birth rate ~ ...
1
vote
1answer
61 views

Gibbs sampling for reducible chain

I am new to Gibbs sampling and I ran into a problem with irreducibility. For the Gibbs sampler to work the Markov chain has to be irreducible. But that assumption is not satisfied in my probability ...
0
votes
1answer
57 views

writing down markov chain transition matrix

Question: An experimental animal can stay in room-A until 1 minute,and it can stay in room-B until 2 minutes. There exist deadly gases in room-C. One room among these three rooms is being randomly ...
2
votes
1answer
32 views

Aperiodicity in markov chain

given this transition matrix of markov chain 1/2 1/4 1/4 0 1/2 1/2 1 0 0 which represents transition matrix of states a,b,c. a has probability of 1/2 ...
0
votes
0answers
16 views

Generating markovian paths

given a 4 by 4 transition matrix P, I want to simulate m Markovian paths of length n I created this function can anyone tell me if this is is correct ? ...
3
votes
1answer
46 views

Gibbs Sampler - Sample mean convergence

To simulate from the posterior distribution $p(\theta|Y)$ where $\theta = (\mu,\lambda_1,\lambda_2)$, I run a Gibbs sampler to draw approximately random values from $p(\theta|Y)$. This Gibbs sampler ...
5
votes
1answer
34 views

Gibbs Sampler output: how many Markov chains?

When running a Gibbs sampler (for $n=200$ Iterations) with two full conditionals, I get the output $\mathbf{x} = (x_1^{(n)},x_2^{(n)})_{n =1,...,200}$. So $\mathbf{x}$ is the realizations of a Gibbs ...
1
vote
0answers
29 views

Appropriate distance measure between two finite state Markov chain models?

I am empirically creating Markov chains similar to this question. I end up with several finite state Markov chain models with the same nodes but varying transition probabilities. I want to calculate a ...
0
votes
0answers
14 views

Q: texts on Markov chain survival analysis

I'm looking for good introductory texts (books, papers,..) on survival analysis using Markov chains. Both theory and application in the medical sciences/epidemiology. I only found the vignette of the ...
0
votes
0answers
31 views

Markov Process w/ a non-stochastic matrix?

I've come across a problem which at first appeared to be a markov process however the transition matrix of the graph is non-stochastic. That is, the probabilities among edges leaving a node do not sum ...