The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [markov-process]

A stochastic process with the property that the future is conditionally independent of the past, given the present.

957 questions
Filter by
Sorted by
Tagged with
101 views

How can we verify the intuition that in the RW-Metropolis-Hastings algorithm with Gaussian proposal too small and too large variances are bad choices

Let $d\in\mathbb N$ and consider the Random Walk Metropolis-Hastings algorithm with a Gaussian proposal kernel $Q$ such that $Q(x,\;\cdot\;)=\mathcal N_d(x,\sigma^2_dI_d)$ for all $x\in\mathbb R^d$. ...
77 views

Relation between Uniform distribution, Metropolis Algorithm, and Symmetric Proposal Distribution

I am having some confusion over the Metropolis algorithm. Let $g(x|y)$ be our proposal distribution for the algorithm. For the Metropolis, $g$ must be symmetric (from Wikipedia). In the discrete case, ...
18 views

Markov Chain Question/Notation Confusion

Show that if $(X_n)_{n \geq 0}$ is a discrete-time Markov chain with transition matrix $P$ and $Y_n = X_{kn}$, then $(Y_n)_{n \geq 0}$ is a Markov chain with transition matrix $P^k$. I am a little ...
119 views

95 views

Transition probability in Markov chain using a decision tree model

I wish to find a way to calculate the transition probabilities in my Markov chain model. Let's say a customer has three products [A B C] and in this example I wish to know the transition probability ...
102 views

31 views

Sequence Prediction with noise / gap using markov models

I'm trying to understand if Markov models can account for a "noise event" when predicting the next item in a sequence. For instance, if i have very frequently occurring (noise) event "F", can a ...
90 views

What is the difference between homogeneous Markov chains and unhomogeneous Markov chains?

I learned that a Markov chain is a graph that describes how the state changes over time, and a homogeneous Markov chain is such a graph that its system dynamic doesn't change. Here the system dynamic ...
98 views

Markov Models for time series prediction

I am student conducting an experiment with different models for time series prediction. In my experiment, I am going to use ARIMA, a Recurrent Neural Network, a Long-Short Term Memory network, and a ...
33 views

Finding Probability of a digit given a sequences?

I have a n sequences of numbers ranges from 1 to 4, say sequence s1 = [1,3, 1, 4] and s2 = [2,1,3,4] up to s(n). My question is how can i find probability of a number coming right after a sequence, ...
44 views

Discrete-time Markov Chain; $n$-step transitions

Let $\{X_{n}\}_{n\geq0}$ be a discrete-time Markow chain on the state space $S=\{1,2,3\}$ with transition matrix \begin{pmatrix} 1/3 & 1/3 & 1/3 \\ 0 & 2/3 & 1/3 \\ 2/3 & 1/...
25 views

Trouble understanding derivation of probability for continuous time markov chain

I'm working on exercise 6.10 from "Introduction to probability models" by Sheldon M. Ross. There's an expression for the probability $P_{00}(t)$ that I don't understand. Here's the relevant ...
31 views

Entropy/Measure of Knowledge for probability intervals

I have a system which outputs probability interval tuples over the decision set $\{d_1, d_2, d_3\}$, such that a tuple $([\min_1, \max_1], [\min_2, \max_2], [\min_3, \max_3])$ indicates that $\min_i$ ...
169 views

Visualization of the number of transitions between states [closed]

I am currently developing a Markov model for ordinal data. In order to proceed with the modeling, I would like to check the distribution of the number of transitions per individual in my data set. ...
59 views

How to use trial and error algorithm to predict the next number in a sequence?

I have a time series data. I want to use trial and error algorithms to predict the next number in a variation_sequence. I mean about Trial and error algorithm is using an online learning and where I ...
27 views

65 views

Biased coins and Markov processes

Good day, I am attempting an optional exercise and I am finding it hard to interpret the problem in terms of matrices and vectors. Coin 1 has probability 0.4 of coming up heads, and coin 2 has ...
34 views

What does environment dynamic means in Reinforcement learning

From the book Reinforcement learning: an introduction, I have two questions: 1) there is the following sentence: "If the environment's dynamics are completely known, then finding the optimal policy ...
132 views

Probabilistic user behavior markov models on web

I am considering the following probabilistic Markov model of actions of a user on the results page of a search engine. The user examines the first result, with a probability $A$ he is satisfied with ...
84 views

States of Markov chain and stationary distribution

Let $X$ be a Markov chain with a state space $S={\{0,1,2,... \}}$ and a transition matrix $P$ with given $p_{i,0}=\frac{i}{i+1}$ and $p_{i,i+1}=\frac{1}{i+1}$, for $i=0,1,2,...$. Find out which states ...
161 views

Proving that given Markov chain is homogeneous. Find state space and transition matrix

Let $X_i$ be the results of a consecutive throws of a die. Let $Z_n=3(X_1^2+\cdots+X_n^2) \bmod 5$. Show that the sequence ${\{Z_n \mid n\geq1\}}$ is a homogeneous Markov Chain. Find a state space and ...
110 views

Expected number of steps in Gambler's ruin game with two players

Let's say we have two players A and B. Player A has 3 coins and player B has 5 coins. If player wins the other player gives one coin. During game second player probability of loosing is $2/3$, while ...
138 views

Why is variational Bayesian mixture model an alternative to MCMC? What are the similarities?

Why do people say that a variational Bayesian mixture model could be an alternative to MCMC for clustering? For example see the details here: https://en.wikipedia.org/wiki/Variational_Bayesian_method. ...
31 views

29 views

State prediction of how long someone sleeps using neural nets

I have over hundred thousands of datapoints on how long individual people sleep. I also have information about how soft their beds are, their income, stress levels etc. At first I want to predicted ...
195 views

What is the difference betwen a time non-homogenous Markov Chain and a non-linear Markov Chain? Example

A time non-homogenous Markov Chain is one in which the transition probabilities are not constant over time. A non-linear Markov Chain is a model that is not linear in parameters and satisfies the ...
28 views

Markov chain inference confusion

I have a Markov chain where: A->B->C->D I am told that P(A,C,D|B) = P(A|B)P(C,D|B) I am unable to prove why this is the case. Why is this as such?