0
$\begingroup$

I'm studying David Silver's second lesson on reinforcement learning: https://www.youtube.com/watch?v=lfHX2hHRMVQ&list=PLqYmG7hTraZDM-OYHWgPebj2MfCFzFObQ&index=2 and the state transition probability matrix is defined as enter image description here

However, with this python code:

import numpy as np

P = np.matrix([[0, 0.5, 0, 0, 0, 0.5, 0], [0, 0, 0.8, 0, 0, 0, 0.2], [0, 0, 0, 0.6, 0.4, 0, 0], \ [0, 0, 0, 0, 0, 0, 1], [0.2, 0.4, 0.4, 0, 0, 0, 0], [0.1, 0, 0, 0, 0, 0.9, 0] ,[0, 0, 0, 0, 0, 0, 1]])

np.linalg.det(P)

I'm getting a determinant of 0 and thus a singular matrix.

The issue is that David then defines the Bellman Function to get the vectorized value functions as: enter image description here

And thus if P can't be inverted, neither can (1-gamma*P) and I can't get V. Any tip on what am I doing wrong? :)

$\endgroup$
9
  • $\begingroup$ Re "and thus:" that's incorrect. For instance, when $\gamma=0$ obviously $1-\gamma P = 1$ is invertible. Indeed, $1-\gamma P$ is invertible except for a small finite number of values of $\gamma.$ $\endgroup$
    – whuber
    Nov 13 '19 at 13:40
  • $\begingroup$ However, in the lesson David shows in the slides v values for gamma=0.9 $\endgroup$ Nov 13 '19 at 13:47
  • 1
    $\begingroup$ There's no problem: $1-0.9P$ is invertible. The only values of $\gamma$ for which $1-\gamma P$ fails to be invertible are $1,$ $1.046604,$ $1.548519,$ $-1.073406\pm2.148352i,$ and $-3.039221.$ These are the reciprocals of the nonzero eigenvalues of $P.$ $\endgroup$
    – whuber
    Nov 13 '19 at 14:17
  • $\begingroup$ Thanks for your reply whuber. As you can see here: ibb.co/x7QH3F6 I'm getting that the determinant of 1-0.9*P is 0. Maybe I'm doing something wrong $\endgroup$ Nov 13 '19 at 14:34
  • $\begingroup$ Help me out with Python here (because I haven't used it in a decade): does 1-P subtract $P$ from the $7\times 7$ identity matrix (ones on the diagonal, zeros elsewhere) or does it perhaps subtract $P$ from the ones matrix (equal to $(1,1,\ldots,1)^\prime\cdot (1,1,\ldots,1)$)? I suspect it's the latter but the former is the intended meaning. $\endgroup$
    – whuber
    Nov 13 '19 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.