In the Compatible Function Approximation Theorem, the following condition is required to make the policy gradient to be exact $\nabla J(\theta) = \mathbb{E}_{\pi_{\theta}}\left [\nabla_{\theta}log\pi_{\theta}\left (s,a\right )Q_{W}\left (s,a\right ) \right ] $.

There is a condition that the value function approximator is compatible to the policy $$\nabla_{W}Q(s,a)=\nabla_{\theta}log\pi_{\theta}(s,a).$$

I understand that for a given policy function $\pi_{\theta}(s,a)$, we can derive its gradient $\nabla_{\theta}log\pi_{\theta}(s,a)$, and multiply parameter $W$ to get the value function $Q(s,a)=\nabla_{\theta}log\pi_{\theta}(s,a) W$.

However, it is more practical to build more expressive value functions, e.g., a deep network; this breaks the compatible condition defined above. As a result, the value function approximator can no longer make the policy gradient exact.

My question is, what are the theoretical properties when the compatibility assumption does not hold? How to analyze this problem?

  • $\begingroup$ what is this condition about ? How can derivative of the Q function generator make equal to the score function ? $\endgroup$ – Shamane Siriwardhana Nov 30 '17 at 11:16
  • $\begingroup$ The condition makes the gradient unbiased. (it does not have anything to do with being "exact") $\endgroup$ – A.D Dec 2 '17 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.