# Tagged Questions

A stochastic process with the property that the future is conditionally independent of the past, given the present.

0answers
27 views

### Randomly generating transition probabilities for Markov chains

I'm trying to simulate a person moving through a household using a Markov chain. Each state would be a room in the house. The issue I'm running into is that I have no existing data telling me what a ...
0answers
30 views

### 'Lumpable states' analysis for a large transition matrix

I have a large transition matrix, whereby I calculate n-step state distribution results for n=1..10, and then merge states of interest for each ...
0answers
23 views

### From MDP to SMDP: What is it in a nutshell

Markov Decision Process (MDP) is a mathematical formulation of decision making. An agent is the decision maker. In the reinforcement learning framework, he is the learner or the decision maker. We ...
0answers
38 views

1answer
40 views

1answer
55 views

### Convergence diagnostic of Markov chain that converge to uniform

Let $\Omega$ be a finite state space, $(X_t)_{t\in\mathbb{N}}$ be a discrete-time Markov chain that converges to the uniform distribution, and $P$ be its transition matrix. I'm looking for different ...
1answer
60 views

### Compute smoothed probabilities for EM algorithm [closed]

In order to compute the expected value of log-likelihood in EM algorithm, we use 3 different probabilities Forecast (predictive) probabilities Inference probabilities Smoothed probabilities ...
0answers
39 views

### How to properly show the efficiency of a process?

I'm no statistician but my background is in computer science. At work, we are trying to improve the efficiency of a system where 5 people (A-E) each produce one part of a report and send it to 2 key ...
2answers
46 views

### Proof that Markov Property is not Satisfied at any Order?

My textbook has this figure in it: The textbook then says, Using d-separation, we can see there is always a path connecting $x_n$ and $x_{m}$ via the latent variables. This makes sense to me because ...
2answers
27 views

### Reinforcment learning MDP optimal policy existence

Reinforcement Learning: An Introduction. Second edition, in progress. Richard S. Sutton and Andrew G. Barto (c) 2012. Solving a reinforcement learning task means, roughly, finding a policy that ...
0answers
31 views

### Modeling the joint distribution of stream statistics

I have a question regarding computing the joint discrete probability distribution of statistics in a number stream. I posted this problem in the Mathematics section as well but I'm hoping the ...
0answers
18 views

### P value for Markov table

I have the following two-state Markov chain: ...
0answers
29 views

### Methodology to calculate Blackjack optimal strategy

Which methodology is best way to derive optimal blackjack strategy for player? I have come across some articles which suggest Markov chain. Is this the best way? Any reference/resource will be ...
1answer
10 views