A stochastic process with the property that the future is conditionally independent of the past, given the present.

learn more… | top users | synonyms (1)

1
vote
0answers
19 views

MLE of CTMC parameters

Let the data set be $$D = \{(s_0, t_0), (s_1, t_1), ..., (s_{N-1}, t_{N-1})\}$$ where $N=|D|$. Each $s_i$ is a state from the state space $S$ and during the time $[t_i,t_{i+1}]$ the chain is in state ...
1
vote
0answers
19 views

How I can use markovchain package when I have many sequences? [closed]

I use R program and markovchain-package. I have file where I row I one person. In row is a list, where countries that person has vited and what order for example Person one: USA, Canada, Germany ...
0
votes
0answers
26 views

Markov Modeling, My repair is constant and not memoryless

I want to calculate a markov model, but there is a problem; my repairing transition is not memoryless and it's constant for every time that it will happen.(Markov model consider all of transitions are ...
6
votes
1answer
86 views

Expected number of times you spent in a state of an absorbing markov chain, given the eventual absorbing state

It's well known that, if $Q$ is the matrix of transient state transition probabilities, and $$ N = \sum_{n=0}^{\infty} Q^n = (I - Q)^{-1}$$ then $N_{ij}$ describes the expected number of times the ...
2
votes
1answer
53 views

Calculating probability for a continuous time markov chain

I can do question 1-3 no problem however I cant follow the steps to get the answer to part 4. Can anyone here explain the steps I need to go through in more detail?
0
votes
0answers
4 views

Measure of incoming communication

Given a standard Markov Chain on discrete time and finite statespace, represented by a matrix $M$, with $\sum_{j=1}^n m_{ij}=1$ I have a certain absorbing state k, where the incoming communication is ...
0
votes
1answer
28 views

how to calculate the following conditional probability

There are two events involved, say event A and event B. I want to know the probability of event B conditioned on the event A. The relation between the two events are as follows. We can not talk about ...
0
votes
0answers
9 views

Texture Synthesis

I'm looking to synthesise some textures that look like natural ceramic tiles in Matlab. This isn't something I've tried before so I was wondering what the best techniques to use would be or if there's ...
3
votes
0answers
197 views

How to understand Gibbs distribution

I have a graph model such as Following the Hammersley–Clifford theorem describes that Markov random fields exhibit a Gibbs distribution with an energy function as follows: $$P(x)=\frac ...
0
votes
0answers
23 views

Calculating joint density function of Brownian motion

I read in my book today regarding the calculation of the joint density function of a brownian motion process and it went as follows: If we define $X(t)$ as a Brownian motion process with mean $0$ and ...
0
votes
0answers
21 views

Transition Probability Matrix with zeros

A markov chain model TPM for daily rainfall is given by: [0.4 0.6 0.0; 0.2 0.6 0.2; 0.0 0.7 0.3]; Thus, it is not possible to pass directly from state 1 to ...
0
votes
0answers
50 views

How can I infer the value of multiple dependent continuous random variables in conjunction with discriminative learners?

I have 2 continuous random variables V1, V2 which are dependent. I want to infer each of their values based on: The value of ...
0
votes
0answers
30 views

Binning by standard deviations

Quick question: I came across a fairly respected source on running Markov Chain Monte Carlo for bayesian statistics in ...
0
votes
0answers
18 views

Estimating transition probabilities from the sequences without repeated states

Is it possible to estimate transition probabilities of the continuous process if we observe only the first occurence of the state? For example, if the real transition sequence was 1 - 5 seconds; 2 - ...
2
votes
0answers
33 views

Markov models that with several active states

Are there any Markov-like models that can have several active states? So say if trying to determine (the chance) when the person will wake up based on two variables (weather and the time the person ...
0
votes
1answer
25 views

Can reinforcement learning be used as path guidance?

I have an online path guidance system that learns from a set of past experiences(trajectories) to provide a guidance for the user on how to cover the given space in the best way, adopting to whatever ...
0
votes
0answers
30 views

anomaly detection with Markov chain

The paper uses a simple technique to detect intrusions in computer systems. I will briefly explain it and ask a question: The paper proposes a simple 1-order Markov chain modelling approach to detect ...
-1
votes
1answer
42 views

markov chain - probability question

Transition matrix has been written like that; $$\mathcal P = \begin{bmatrix} 1/3 & 0 & 2/3 \\ 1/3 & 1/3 & 1/3 \\ 0 & 0 & 1 \end{bmatrix}$$ the initial vector is that ...
1
vote
0answers
22 views

How to estimate Markov chain transition probabilities with partially observed data?

Suppose that we have a time-homogeneous discrete-time Markov chain $(X_n)$. We want to estimate the transition probabilities $p_{ij} = \mathbb{P}[X_{n+1} = j \mid X_n = i]$. In the case when we have ...
1
vote
0answers
40 views

Measuring effectiveness of marketing through attribution analysis [closed]

My data(dataframe in R) looks like this:The data is ordered by CustomerName and then TimeofEvent. ...
3
votes
0answers
38 views

Markov decision process in R for a song suggestion software?

We have a music player that has different playlists and automatically suggests songs from the current playlist I'm in. What I want the program to learn is, that if I skip the song, it should decrease ...
0
votes
0answers
5 views

Identifying sequences of behavioral interactions between multiple individuals

I'm wondering if anyone might have some novel insights as to the best way to analyze the following data. It's a problem I've been thinking about in the back of my mind for a while, so I thought that ...
2
votes
1answer
65 views

Markov chains with a stationary distribution but no limiting distribution

I am trying to intuitively reconcile the following statement, read from "Probability, Markov Chains, and Queues": ...
2
votes
1answer
34 views

First-order Discrete Markov Chain with time lag

I want to estimate the first-order transition matrix of a sequence in discrete time, e.g. $$ s = 1,0,1,0,1,1,0,1,0,0, \dots$$ but states are not evenly spaced in time. So that even if $s_{t=1} = 1$ ...
0
votes
0answers
7 views

Estimate conditional probability of connected variables

I want to model the following probability function $p(x_i|\mathcal{N}_{x_i})$, where $\mathcal{N}_{x_i}$ is the set of the variables $x_j$ conneced to $x_i$ given a specified undirected graph ...
0
votes
0answers
15 views

Degrees of Freedom for Inhomogeneous Markov Chains

Suppose I am given a time-inhomogeneous Markov chain. For simplicity, let's assume we have 5 (time) positions, and 4 states (A,B,C,D). How can I compute the degrees of freedom for a first order model ...
0
votes
1answer
42 views

What is the expected amount of time it takes to first observe a triple for multiple dice rolls? [duplicate]

Assuming we have a fair die, we toss a die multiple times. Also assuming that a triple is defined as when we have three rolls in a roll that result in the same number, and that the rolls are ...
4
votes
1answer
49 views

Analyzing output in MCMC

I am using emcee to do inference on some data. I am trying to fit my data to a line of equation $ y = mx + b $. ...
4
votes
1answer
44 views

Given two absorbing Markov chains, what is the probability that one will terminate before the other?

I have two different Markov chains, each with one absorbing state and a known starting position. I want to determine the probability that chain 1 will reach an absorbing state in fewer steps than ...
0
votes
0answers
25 views

Limiting distribution of a Markov chain?

I have the problem below. There are n identical machines. They are all operational at time 0. The lifetime of each one is an exponential random variable with rate L. There are r repairmen (1 ≤ r ≤ ...
0
votes
0answers
11 views

What are good tutorial on Weighted Finite Automata?

I would especially appreciate papers, books or tutorials with source code already available. Currently I'm reading "Spectral Learning Techniques for Weighted Automata, Transducers, and Grammars" by ...
1
vote
1answer
28 views

Estimating Markov Switching Probit

I attempt to fit the following probit model to a time series where we observe the binary variable $R_{t}$ and another variable $X_{t}$, a latent unobserved variable $y^{*}_{t}$ and a state variable ...
-1
votes
1answer
22 views

Proof of Markov Chain property

Suppose that $X_n$ is a Markov Chain.Then for $m,n \in N$ such that $m<n$ $Pr[X_n=j_n|X_m=j_m,X_{m-1}=j_{m-1},...=X_0=j_0]=Pr[X_n=j_n|X_m=j_m]$ When proving for n=3,m=1 case we have to show ...
1
vote
1answer
18 views

Conditional transition matrix (consistent estimates)

Is there a model/technique that is able to estimate transition matrix (which would be consistent, i.e. sums of their rows would be always 1) conditional on some continuous variable X? Let's say I ...
0
votes
0answers
18 views

Distribution of the duration of a markov-process in a specified state during a specified time

I have a continuous time markov chain with two states $A$ and $B$. The transition rate $A\rightarrow B$ is $\lambda$ and $B\rightarrow A$ is $\mu$. Imagine that $P(X{t_0}=A)=1$ (the process starts in ...
0
votes
0answers
17 views

disease progression R markov chains

hello i have a dataframe, some of the columns include: remission,height,weight,time from diagnosis, age ethnicity, age, patient id remission is 1 or 0 just to be clear i want to fit an appropriate ...
0
votes
0answers
38 views

Subsetting dataframe conditions on factor (binary) column (vector in R)

I have a sequence of 1/0's indicating if patient is in remission or not. Assume the records of remission or not were taken at discrete times. How can I check the Markov property for each patient, ...
1
vote
1answer
32 views

r markov chain markov property on binary variable, discrete time

i have a sequence of 1/0's indicating if patient is in remission or not, assume the records of remission or not were taken at discrete times, how can i check the markov property for each patient, ...
1
vote
1answer
112 views

Fit and evaluate a second order transition matrix (Markov Process) in R?

I already built 1 first order discrete state Markov Chain model. It was built with R using the function 'markovchainFit()' in ...
0
votes
0answers
19 views

N-gram learning vs stochastic learning

I'm interested in comparing the differences in learning in n-grams and gradient-based learning (in my case with neural networks), particularly in the context of language modelling with the two classes ...
0
votes
0answers
25 views

Markov chains hitting times

I'm having trouble understanding what hitting times are in markov chain processes and how they are calculated. An example follows: A Markov process on E = {1, 2, 3} has the following generator matrix ...
0
votes
1answer
40 views

Markov Process-Variance of time until jump

A Markov process on E = {1, 2} is constructed according to holding time parameters λ1 = 2 and λ2 = 4; the defining Markov chain has transition probabilities p11 = p12 = 0.5 and p21 = 1. How do I ...
0
votes
1answer
26 views

Finding One Step Transition Matrix in Gambling?

I need help finding what a one step transition matrix would look like for the following gambling scenario: Using the bold strategy, say you have a certain amount of money x at any time and you're ...
-1
votes
1answer
27 views

Does the order of variables in a Markov Regime Switching model matter?

since Ive received feedback that my previous question was not well-recieved Ill just have to give it another shot. I am estimating Markov Regime Switching Models, and I am getting different results ...
1
vote
0answers
26 views

How do you show that a Markov chain has not mixed?

I came across the claim that when doing Gibbs sampling, it is possible to show that a Markov chain has not yet mixed. How is this done?
2
votes
2answers
35 views

Distribution of p(x) in empirical model

I am having a hard time to exactly name what I am looking for (I am quite sure it already exists out there...) so I'll start with a concrete example: I have a population of discrete colours (red, ...
8
votes
1answer
90 views

Probability of a consecutive pair of values

Lets $X=(x_1, x_2,...x_{20})$ where $x_i\sim N(0,1)$ and $x_i, x_j$ are independent $\forall i\neq j$. What is the probability to obtain a sample $X$ where there are at least two consecutive values ...
3
votes
0answers
80 views

simulating birth death process with random numbers from negative binomial

I am trying to generate random deviates for the population size at time $t$ for a birth-death process with constant birth and death rates per individual and initial size $N_0 \gt 0$. For the simple ...
0
votes
0answers
27 views

Real-life examples of Markov Decision Processes

I've been watching a lot of tutorial videos and they are look the same. This one for example: https://www.youtube.com/watch?v=ip4iSMRW5X4 They explain states, actions and probabilities which are ...
0
votes
1answer
18 views

Birth & Death process - Combining Transition rates

I think I'm missing a fundamental step in regards to how to combine two exponential distributions in the context of this problem. If we have a birth and death process where birth rate ~ ...