Policy Gradient

Basic Idea

$\bar{v}_\pi = \sum_s d_\pi(s) v_\pi(s)$

$\bar{r}_\pi = \sum_s d_\pi(s) \sum_a \pi(a|s) r(s, a)$

$\bar{r}_\pi = \lim_{T \to \infty} \frac{1}{T} \mathbb{E}\left[\sum_{t=1}^{T} r_t\right]$

$\nabla_\theta \bar{v}_\pi = \sum_s d_\pi(s) \sum_a \nabla_\theta \pi_\theta(a|s) q_\pi(s, a)$

$\nabla_\theta \bar{r}_\pi = \sum_s d_\pi(s) \sum_a \nabla_\theta \pi_\theta(a|s) q_\pi(s, a)$

利用梯度上升算法 (Gradient Ascent), 最大化长期奖励 (learn from rewards and mistakes):

Gradient Ascent

\begin{equation} \begin{split} \theta^*&=\arg\max\limits_\theta\bar{R}_\theta=\arg\max\limits_\theta\sum\limits_{\tau}R(\tau)P(\tau|\theta)\\ \theta_{t+1}&=\theta_t+\eta\nabla\bar{R}_\theta\\ \nabla\bar{R}_\theta&=\begin{bmatrix}\frac{\partial\bar{R}_\theta}{\partial{w_1}}\\\frac{\partial\bar{R}_\theta}{\partial{w_2}}\\\vdots\\\frac{\partial\bar{R}_\theta}{\partial{b_1}}\\\vdots\end{bmatrix}\\ R_t&=\sum\limits_{n=t}^N\gamma^{n-t}r_n \end{split} \end{equation}

$\theta \leftarrow \theta + \alpha G_t \nabla_\theta \ln \pi_\theta(a_t|s_t)$

$\theta \leftarrow \theta + \alpha (G_t - b(s_t)) \nabla_\theta \ln \pi_\theta(a_t|s_t)$