# Singular state transition probability matrix in David Silver's UCL Lesson 2

I'm studying David Silver's second lesson on reinforcement learning: https://www.youtube.com/watch?v=lfHX2hHRMVQ&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&index=2 and the state transition probability matrix is defined as However, with this python code:

import numpy as np

P = np.matrix([[0, 0.5, 0, 0, 0, 0.5, 0], [0, 0, 0.8, 0, 0, 0, 0.2], [0, 0, 0, 0.6, 0.4, 0, 0], \ [0, 0, 0, 0, 0, 0, 1], [0.2, 0.4, 0.4, 0, 0, 0, 0], [0.1, 0, 0, 0, 0, 0.9, 0] ,[0, 0, 0, 0, 0, 0, 1]])

np.linalg.det(P)

I'm getting a determinant of 0 and thus a singular matrix.

The issue is that David then defines the Bellman Function to get the vectorized value functions as: And thus if P can't be inverted, neither can (1-gamma*P) and I can't get V. Any tip on what am I doing wrong? :)

• Re "and thus:" that's incorrect. For instance, when $\gamma=0$ obviously $1-\gamma P = 1$ is invertible. Indeed, $1-\gamma P$ is invertible except for a small finite number of values of $\gamma.$
– whuber
Nov 13 '19 at 13:40
• However, in the lesson David shows in the slides v values for gamma=0.9 Nov 13 '19 at 13:47
• There's no problem: $1-0.9P$ is invertible. The only values of $\gamma$ for which $1-\gamma P$ fails to be invertible are $1,$ $1.046604,$ $1.548519,$ $-1.073406\pm2.148352i,$ and $-3.039221.$ These are the reciprocals of the nonzero eigenvalues of $P.$
– whuber
Nov 13 '19 at 14:17
• Thanks for your reply whuber. As you can see here: ibb.co/x7QH3F6 I'm getting that the determinant of 1-0.9*P is 0. Maybe I'm doing something wrong Nov 13 '19 at 14:34
• Help me out with Python here (because I haven't used it in a decade): does 1-P subtract $P$ from the $7\times 7$ identity matrix (ones on the diagonal, zeros elsewhere) or does it perhaps subtract $P$ from the ones matrix (equal to $(1,1,\ldots,1)^\prime\cdot (1,1,\ldots,1)$)? I suspect it's the latter but the former is the intended meaning.
– whuber
Nov 13 '19 at 15:26