3
$\begingroup$

The above model is from Berkeley cs294-112 Page 23. It is said that $$ p(O_{1:T},s_{1:T}, a_{1:T}) =p(s_1)\prod_{t}p(s_{t+1}|s_t,a_t)p(O_t|s_t,a_t)\tag 1 $$

I'm quite confused about this solution: where is $p(a_t|s_t)$?

I'm thinking whether this has something to do with the assumption that $p(a_t|s_t)$ is a uniform distribution, but it uses $=$ instead of $\propto$ and the result is further used in the computation of the Evidence Lower Bound. How is Eq.(1) derived?

$\endgroup$
  • $\begingroup$ can you point us to the page you saw this formula? $\endgroup$ – gunes Feb 7 '19 at 8:58
  • $\begingroup$ @gunes It is in Page 23 $\endgroup$ – Maybe Feb 7 '19 at 13:02
0
$\begingroup$

It seems they assume (for simplicity) that $ p(a_t|s_t) $ is uniform (they explicitly say so in page 9 (Backward messages)), as you said, and therefore its density function is $ f(a_t|s_t) = 1 $ (they assumed standard uniform), and since it is 1 it disappears from the formula.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I know that $p(a_t|s_t)$ is uniform, but what is $f(a_t|s_t)$? $\endgroup$ – Maybe Feb 7 '19 at 11:09
  • $\begingroup$ If $ a_t $ is a continuous random variable, then $ p(a_t) $ is always 0. It is common in papers to write $ p(x) $ as a general notation, that refers to the probability if it is discrete and to density ($f(x)$) if it is continuous. In this case since it is continuous, by $p$ they actually mean $f$ (density function). $\endgroup$ – Orielno Feb 7 '19 at 18:53
  • $\begingroup$ Hi, @Orielno. Thanks for your clarification. But I still do not understand why you said that the density function of a uniform distribution was 1? What do you think $p(O_{1:T}, s_{1:T}, a_{1:T})$ would be if we did not ignore anything? $\endgroup$ – Maybe Feb 13 '19 at 8:49
  • $\begingroup$ Then it would be as you said, we would add $ p(a_t | s_t) $ $\endgroup$ – Orielno Feb 14 '19 at 11:03
0
$\begingroup$

This is the solution I can get for now, welcome to comment.

The probability model is $$ p(O_{1:T},s_{1:T},a)=p(s_1)\prod_tp(s_{t+1}|s_t,a_t)p(O_t|s_t,a_t)p(a_t|s_t) $$ taking logarithm, we have $$ \log p(O_{1:T},s_{1:T},a)=\log p(s_1)+\sum_t\log p(s_{t+1}|s_t,a_t)+\log p(O_t|s_t,a_t)+\log p(a_t|s_t) $$ because $p(a_t|s_t)$ is uniform, the last term is a constant term. This means it does not contribute to the discussion about the ELBO, so the instructor omitted it for simplicity.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.