Why gradient descent of log probabilities in Alpha Go? In the original AlphaGo paper, it is stated that the policy network is trained with the following gradient:



I don't understand why this gradient makes sense. Why do we want to move the parameters in the direction where the log probabilities would be higher (in case of success) or lower (in case of fail)?
 A: The proof that this is the correct gradient to use in order to improve utility from following a policy function is called the Policy Gradient Theorem.
In brief, in that derivation, the $\text{log}$ function appears due to a division by the probability of taking the action (i.e. the same policy function). This occurs when writing out the gradient of the utility of the policy as an expectation, which you wish to do, so you can take samples based on that expectation, and use them to estimate the gradient. Rather than write $\frac{\nabla x}{x}$ you can write $\nabla \text{log}(x)$.
In addition, as the gradient is scaled by $z_t$ (the value of the outcome), then gradients will be larger and positive when the probability of taking an action was low, but the outcome was good - increasing the probability here in the update step will be good for future outcomes. Whilst if $z_t$ is negative (the game was eventually lost), probabilities of the chosen actions will be reduced.
Often you also want to offset $z_t$ factor by some baseline measurement (such as average reward, or current estimated value of a state, or use the "advantage" function instead of state value) to stabilise a policy gradient algorithm - else it can become a race between relative values with the preferences for all actions becoming higher and higher, and with better ones increasing faster (until non-favourable outcomes are so rare they stop happening anyway). With the simple reward structure and zero-sum maths of a two-player game, this seems not to be necessary for AlphaGo.
A: Note that actions we take are determined by weights of a CNN. If we win, then it means that actions we took are correct, which means that we want our actor to behave similarly in the future. What it means formally, though, is that probability of sampling such actions should be high. We make it happen by moving our weights in that directions. 
If we lose, then it means that in this situation actions we took were bad. Hence we prefer to avoid sampling actions like this - formally it means that we ascribe low probability to them. 
