Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

Filter by
Sorted by
Tagged with
2
votes
0answers
96 views

Markov random field and iterated condition mode

I have spent a lot of time studying MRF (applied to images) but still can't grasp the idea. Could you please clarify these ideas: What is the clique potential? What is a clique in image, and do they ...
3
votes
1answer
401 views

Stationary matrix given a transition matrix

I am given the following transition matrix $$P= \pmatrix{ 1-\alpha & \alpha \\ \beta & 1-\beta}, \ \alpha,\beta \in (0,1)$$ with the states $S=\{1,2\}$. I want to determine the stationary ...
3
votes
1answer
438 views

What is “symmetric property” for stationary distribution

I have the one step transition matrix $$\pmatrix{0 & \alpha & 0 & \beta \\ \alpha & 0 & \beta & 0 \\ 0 & \beta & 0 & \alpha \\ \beta & 0 & \alpha & 0 \\...
1
vote
0answers
237 views

Where does the summation sign come from when working out expected value of time

The lifetime of a machine is modelled by an exponential random variable $X$ with $P(X>x) = e^{-\lambda x}, \lambda, x > 0$. This machine cannot be repaired. A maintenance crew checks this ...
1
vote
0answers
686 views

Don't understand how they've calculated the transition probabilities

Suppose the weather condition classified as "sunny" (S) or "rainy" (R). Assume the condition on day $n + 1$ is dependent on the days $n$ and $n-1$ only. Work out the transition matrix from the table ...
3
votes
0answers
91 views

Is there any way to define a distance metric given a Hidden Markov Model?

Let's say I've gotten a HMM that describes user search strings for my e-commerce website. Let's also say that I've just received a search string from a customer that doesn't have any search results. ...
4
votes
1answer
8k views

Solving the Kolmogorov forward equation for transition probabilities

Let $\lambda \mu > 0$ and let $X$ be a Markov chain on $\{1,2\}$ with generators $$ Q = \begin{pmatrix} -\mu & \mu \\ \lambda & -\lambda \end{pmatrix}$$ Write down the forward equations ...
0
votes
0answers
269 views

How does integrating the Kolmogorov forward equation give $P = \exp (Qt)$?

If $Q$ is a generator matrix of a continuous time Markov chain (CTMC), and I need to use this matrix to solve the Kolmogorov forward equation, I would need to start by integrating it. But I haven't ...
1
vote
1answer
153 views

confusion related to higher order markov chain

I was reading this book related to machine learning. It's given that for Mth order markov chain, the number of parameters = $K^{M-1}(K-1)$ where M is the order. I am not sure how this is derived. For ...
2
votes
1answer
1k views

Number of parameters in a Markov chain

I was reading the book Pattern Recognition and Machine Learning by Bishop, which stated that for the first order Markov chain with K states, the number of parameters is K(K-1). Why is that? I think ...
3
votes
2answers
112 views

Licenses renewals prediction

I have come recently to the following real world problem concerning licence renewals of a software product. I have just rudimentary knowledge of the basics in this field and I mostly interested in ...
4
votes
1answer
340 views

Tips on good literature regarding Markov chains and processes

I have read a course in Markov processes at my uni (Im a graduate student in Lund, Sweden) and would like to dig a bit deeper into the field. The book provided for that course was written in by a ...
0
votes
2answers
357 views

Do hidden Markov models contain Markov chains?

Is it correct to say that the Hidden State Sequence in a Hidden Markov Model is a Markov Chain? Thanks
2
votes
1answer
98 views

computing transition probabilites given only aggregate counts

I asked something like this on math.stackexchange but it got no answers so I'm hoping for more enlightenment here (the other question is this one). So: We have some number of people; each person i is ...
2
votes
1answer
6k views

First-order Markov chain, states transition probabilities at each time are enough for the model

Hi markov chain specialist, Hope you can give me an answer regarding this trellis diagram that i saw on a book. why in this picture of a general first-order markov chain of 2 states,we should know ...
2
votes
0answers
109 views

Metric for ordinal, non-normal distribution in Markov models

I am working with Markov models with ordinal, non-normal distributions of probabilities. Ultimately, I would like to create a metric for determining the probability of a specific path occurring. ...
4
votes
1answer
621 views

R packages or open source software for training Hidden Markov chains

Are there any well-designed R packages or other open-source software for training Hidden Markov chains?
5
votes
2answers
502 views

MRF MAP inference for non-submodular pairwise terms

I have a multilabel MRF MAP inference problem (a labeling problem). The graph has relatively few nodes, about a thousand or so. The pairwise term is (very) not submodular (it does not satisfy the ...
24
votes
1answer
1k views

Random matrices with constraints on row and column length

I need to generate random non-square matrices with $R$ rows and $C$ columns, elements randomly distributed with mean = 0, and constrained such that the length (L2 norm) of each row is $1$ and the ...
4
votes
2answers
456 views

Confusion related to MCMC technique

I have this confusion related to Monte Carlo Markov Chain method. I know that Monte Carlo method is used to get the sample mean instead of calculating the high dimensional integration which is not ...
5
votes
1answer
2k views

Probability of visiting all other states before return

Question (a) Random walk on a clock. Consider the numbers $1, 2, \dots, 12$ written around a clock. Consider a Markov chain that jumps with equal probability to one of the two adjacent numbers each ...
5
votes
3answers
3k views

Identify different periods of variance in a time series

I have a time series $x_t$ which may go through different phases of volatility. One example might be some stock that has high variance from 9 AM to 11 AM, low variance from 11 AM to 2 PM, and then ...
3
votes
1answer
890 views

Proving a non-stopping time

Let me begin by first confirming that this is indeed the correct place to post this (other ideas I had were math.SE). That said, Let $X_n$ be a Markov chain on the state space $\mathcal S$ and for $ ...
2
votes
0answers
125 views

In a hidden Markov model, how do all observations and one state give you all states?

$Y^n$ are the observations of our HMM, where $Y_i=a_i$ is a single observation, where $a_i \epsilon \{0,1\}$. For example, $Y^n = k^n$ where $k^n=\{0,1,1,0\}$ $X^n$ are the actual states of our HMM, ...
2
votes
0answers
261 views

Inferring transition matrices in continuous time Markov processes

Suppose I have a process X1, for which I do not have a generator matrix, only a transition (probability) matrix P1 for some time interval T, e.g. T=100. Suppose I have another process X2, such that X2 ...
4
votes
1answer
151 views

Estimating an unknown restricted Markov Chain from partial measurements

There is an Markov chain $M$ defined on states $1, ..., N$ with the special property that it only has transitions $p_i$ from $i$ to $i + 1$ , $q_{i + 1}$ from $i + 1$ to $i$, and $r_i = 1 - p_i - q_i$ ...
2
votes
0answers
156 views

Confusion regarding CRF

I was reading this paper related to conditional random fields http://www.inference.phy.cam.ac.uk/hmw26/papers/crf_intro.pdf. However, I have some confusion related to the section CRF probability as ...
5
votes
2answers
172 views

Confusion regarding random walk model

I was referring to this book where it is given that If we assume equally spaced nodes $i$ for $i=1,...,n$. The first order random walk is constructed using independent increments $$ \Delta{x_i} \...
4
votes
1answer
229 views

Sampler method to choose in Monte Carlo Markov chain estimation

When estimating the posterior via MCMC, are there guidelines on the best sampling method to use depending on the nature of the model? There are a variety of forms of MCMC - the Gibbs sampler, the ...
10
votes
1answer
3k views

Test for markov-property in a time-series

Given an (observed) time-series $X_t$ with $X_t\in\{1,...,n\}$, is there a statistical test for testing the null-hypothesis that $P(X_t|X_{t-1},X_{t-2},...,X_1)=P(X_t|X_{t-1})$ (i.e. the markov-...
2
votes
1answer
664 views

Finding the generator matrix for a Markov jump process

Vehicles in a certain country are required to be assessed every year for road-worthiness. At one vehicle assessment center, drivers wait for an average of 15 minutes before the road-worthiness ...
5
votes
1answer
1k views

Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are $\phi$-...
5
votes
2answers
2k views

Are two empirically estimated Markov chains statistically different?

I am constructing Markov chains (with 100 to 200 states) and inferring transition probabilities empirically by simply counting how many times I saw each transition in my raw data (about 20k to 60k ...
6
votes
1answer
3k views

Difficulty in understanding Hidden Markov Model for syntax parsing using Viterbi algorithm

I intend to apply Kevin Murphy's Hidden markov model (HMM) toolbox. I have a set of production rules(arbitrary) $A_0 \to AB [p=1]$, $A\to aC [p=1]$, $B\to bbC [p=0.5]$, $B\to b [p=0.5]$ where $A_0$ is ...
5
votes
1answer
2k views

What is the probability of rolling all faces of a die after n number of rolls

It is fairly easy to figure out what is the average number of rolls it would take to roll all faces of a die [$1 + 6/4 + 6/4 + 6/3 + 6/2 + 6/1 = 14.7$], but that got me thinking of a seemingly more ...
3
votes
2answers
7k views

Problem in discrete valued time series forecasting

I have a temporally ordered discrete valued data. The only possible states for the data are: {1,2,3,4,5,6}. So the series is something like {1,2,3,5,6,4,3,5,2,......} I want to forecast the next value ...
4
votes
1answer
504 views

Modeling null transitions in the Hidden Markov Model for use with the Viterbi algorithm

I've implemented the classic HMM model from Rabiner's tutorial for gesture recognition and it has worked well. Now, I'm trying to implement the HMM Threshold Model which calls for an HMM with null ...
4
votes
3answers
764 views

How to sample natural numbers, such that the sum is equal to a constant?

Say I have $N$ items that are partitioned / clustered and I want to randomly repartition these items, such that the distribution of sizes of the clusters is 'similar' to those that I already have. I'm ...
10
votes
3answers
6k views

Expected number of coin tosses to get N consecutive, given M consecutive

Interviewstreet had their second CodeSprint in January that included the question below. The programmatic answer is posted but doesn't include a statistical explanation. (You can see the original ...
2
votes
2answers
4k views

Markov chain long run probabilities

So I want to find the long run probability of getting two heads in a row when flipping a fair coin many times. I know this answer should be 1/6 (expected number of flips to get 2H in a row is 6) but I'...
2
votes
1answer
398 views

Stationary distribution on a partition of the state space?

I would like to use the following type of model and am wondering if someone could let me know if this technique has been studied before (I assume it has) and where to learn more about it? The idea is ...
1
vote
1answer
106 views

Statistically back-calculating: Markov Chain?

I would like to calculate the value of bacteria on 4 surfaces $i=\{1..4\}$. A person touches some of those 4 surfaces at random and a count is made on their finger after each surface contact ($x_i$). ...
0
votes
0answers
109 views

having trouble applying hidden markov models to my game [duplicate]

Possible Duplicate: having trouble applying hidden markov/machine learning models Happy New Year! I’m having a problem applying hidden Markov models to a game I’m building to learn about ...
0
votes
1answer
1k views

Steady state probabilities for a continuous-time Markov chain

I have a finite state and time-homogeneous continuous-time Markov chain (CTMC) which is not irreducible. Will steady state probabilities exist for this CTMC? How to prove this?
5
votes
1answer
350 views

Lumping in Markov process with absorbing states

I have a four-state, discrete time Markov process with time-dependent transition matrices such that after a given time T the matrices become constant. The idea is people in a program leaving the ...
3
votes
0answers
248 views

Summing values of state transitions accumulated in an absorbing Markov chain

I am trying to simulate a process as an absorbing Markov chain model, but I haven't been able to find the scenario that I am interested in looking at in the typical discussions of Markov chains online....
1
vote
2answers
467 views

What methods of statistical analysis can be used for time series data?

I have done many 1-sample T-tests before, but I can't figure out if I am able to use one in this situation. In our experiment, we took 12 individual insects and placed them in a chamber where they ...
6
votes
1answer
2k views

Maximum likelihood estimation procedures for state-space linear models

State-space models are represented by a state equation and an observation equation (or system of equations to be more precise). These equations are parametarized by components including a transition ...
5
votes
1answer
982 views

How can I generate correlated timeseries made up of 0s and 1s?

I want to generate series of 0s and 1s that exhibit some clustering. By this I mean that 1s and 0s should occur together. So I envisage series of 0s and 1s that will exhibit similar clustering of ...
50
votes
3answers
2k views

Do we have a problem of “pity upvotes”?

I know, this may sound like it is off-topic, but hear me out. At Stack Overflow and here we get votes on posts, this is all stored in a tabular form. E.g.: post id voter id vote type ...