在阅读本文前，需要了解 强化学习的基本概念 。本文按顺序探讨了以下几个部分：
策略方法的目标
朴素策略梯度的计算
策略网络的简易训练步骤
reward-to-go 策略梯度
一般形式的策略梯度
在深度学习的背景下，我们规定策略函数使用神经网络近似，表示为：π θ ( a ∣ s ) \pi_{\theta}(a \mid s) π θ ( a ∣ s ) θ \theta θ
本文主要参考 OpenAI Spinning Up Documentation Part 3: Intro to Policy Optimization 编写，对部分内容的进行了删减，感兴趣的读者可以阅读原文。此外，为了确保数学公式的严谨性，参考了 Policy Gradient Algorithms | Lil’Log ，该文章主要围绕数学理论展开。类似的文章还有 Going Deeper Into Reinforcement Learning: Fundamentals of Policy Gradients 。
Optimization Objective
回顾即时收益与回报的定义：
G t = R t + γ R t + 1 + γ 2 R t + 2 + ⋯ = ∑ k = 0 ∞ γ k R t + k G_{t} = R_{t} + \gamma R_{t+1} + \gamma^{2} R_{t+2} + \cdots = \sum_{k=0}^{\infty} \gamma^{k} R_{t+k}
G t = R t + γ R t + 1 + γ 2 R t + 2 + ⋯ = k = 0 ∑ ∞ γ k R t + k
其中：
G t G_{t} G t t t t γ \gamma γ R t = R ( s t , a t , s t + 1 ) R_{t} = R(s_{t}, a_{t}, s_{t+1}) R t = R ( s t , a t , s t + 1 ) t t t R R R s t s_{t} s t a t a_{t} a t s t + 1 s_{t+1} s t + 1
为了简化后续策略梯度的分析，我们从轨迹回报入手。首先，定义长度为 T T T τ = ( s 0 , a 0 , … , s T ) \tau = (s_{0},a_{0},\ldots,s_{T}) τ = ( s 0 , a 0 , … , s T ) τ \tau τ
G ( τ ) = ∑ t = 0 T − 1 γ t R t G(\tau)=\sum_{t=0}^{T-1} \gamma^{t} R_{t}
G ( τ ) = t = 0 ∑ T − 1 γ t R t
强化学习的目标是找到一个策略（Policy），使得 Agent 在与环境（Environment）交互时，能够最大化期望回报（Expected Return）。策略梯度方法的优化目标可定义如下：
max J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ G ( τ ) ] \max J(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}[G(\tau)]
max J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ G ( τ )]
其中，P ( ⋅ ; θ ) \mathbb{P}(\cdot;\theta) P ( ⋅ ; θ ) τ \tau τ
P ( τ ; θ ) = ρ 0 ( s 0 ) ∏ t = 0 T − 1 π θ ( a t ∣ s t ) ⋅ P ( s t + 1 ∣ s t , a t ) \mathbb{P}(\tau;\theta)=\rho_{0}(s_{0}) \prod_{t=0}^{T-1} \pi_{\theta}(a_{t} \mid s_{t}) \cdot P(s_{t+1} \mid s_{t},a_{t})
P ( τ ; θ ) = ρ 0 ( s 0 ) t = 0 ∏ T − 1 π θ ( a t ∣ s t ) ⋅ P ( s t + 1 ∣ s t , a t )
其中：
ρ 0 ( s 0 ) \rho_{0}(s_{0}) ρ 0 ( s 0 ) s 0 s_{0} s 0 π θ ( a t ∣ s t ) \pi_{\theta}(a_{t} \mid s_{t}) π θ ( a t ∣ s t ) s t s_{t} s t a t a_{t} a t P ( s t + 1 ∣ s t , a t ) P(s_{t+1} \mid s_{t},a_{t}) P ( s t + 1 ∣ s t , a t ) s t s_{t} s t a t a_{t} a t s t + 1 s_{t+1} s t + 1
至此，我们已完成对 Policy-based 方法优化目标的定义。接着，可以通过梯度上升法对策略进行优化：
θ k + 1 = θ k + α ∇ θ J ( θ ) ∣ θ = θ k \theta_{k+1} = \theta_{k} + \alpha \nabla_{\theta}J(\theta)|_{\theta=\theta_{k}}
θ k + 1 = θ k + α ∇ θ J ( θ ) ∣ θ = θ k
其中，∇ θ J ( θ ) \nabla_{\theta}J(\theta) ∇ θ J ( θ ) α \alpha α
Naive Policy Gradient
为了便于后续策略梯度的计算，我们首先计算轨迹概率的梯度。
第一步 ，利用对数导数技巧 ∇ x ln x = 1 / x \nabla_{x} \ln x = 1/x ∇ x ln x = 1/ x
∇ θ P ( τ ; θ ) = P ( τ ; θ ) ∇ θ ln P ( τ ; θ ) \nabla_{\theta} \mathbb{P}(\tau;\theta)=\mathbb{P}(\tau;\theta) \nabla_{\theta} \ln \mathbb{P}(\tau;\theta)
∇ θ P ( τ ; θ ) = P ( τ ; θ ) ∇ θ ln P ( τ ; θ )
上式从右往左推导便显而易见。一些教程中将自然对数函数 ln x \ln x ln x log x \log x log x
第二步 ，计算轨迹 τ \tau τ
ln P ( τ ; θ ) = ln ρ 0 ( s 0 ) + ∑ t = 0 T − 1 ln π θ ( a t ∣ s t ) + ∑ t = 0 T − 1 P ( s t + 1 ∣ s t , a t ) \ln \mathbb{P}(\tau;\theta)=\ln \rho_{0}(s_{0}) + \sum_{t=0}^{T-1} \ln \pi_{\theta}(a_{t} \mid s_{t}) + \sum_{t=0}^{T-1} P(s_{t+1} \mid s_{t},a_{t})
ln P ( τ ; θ ) = ln ρ 0 ( s 0 ) + t = 0 ∑ T − 1 ln π θ ( a t ∣ s t ) + t = 0 ∑ T − 1 P ( s t + 1 ∣ s t , a t )
由于环境不依赖于策略网络参数 θ \theta θ ρ 0 ( s 0 ) \rho_{0}(s_{0}) ρ 0 ( s 0 ) P ( s t + 1 ∣ s t , a t ) P(s_{t+1} \mid s_{t},a_{t}) P ( s t + 1 ∣ s t , a t ) G ( τ ) G(\tau) G ( τ ) 0 0 0
第三步 ，计算轨迹 τ \tau τ
∇ θ ln P ( τ ; θ ) = ∇ θ ln ρ 0 ( s 0 ) + ∑ t = 0 T − 1 ∇ θ ln π θ ( a t ∣ s t ) + ∑ t = 0 T − 1 ∇ θ P ( s t + 1 ∣ s t , a t ) = ∑ t = 0 T − 1 ∇ θ ln π θ ( a t ∣ s t ) \begin{aligned}
\nabla_{\theta} \ln \mathbb{P}(\tau;\theta)&=\cancel{\nabla_{\theta} \ln \rho_{0}(s_{0})} + \sum_{t=0}^{T-1} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t}) + \sum_{t=0}^{T-1} \cancel{\nabla_{\theta} P(s_{t+1} \mid s_{t},a_{t})} \\
&= \sum_{t=0}^{T-1} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t})
\end{aligned}
∇ θ ln P ( τ ; θ ) = ∇ θ ln ρ 0 ( s 0 ) + t = 0 ∑ T − 1 ∇ θ ln π θ ( a t ∣ s t ) + t = 0 ∑ T − 1 ∇ θ P ( s t + 1 ∣ s t , a t ) = t = 0 ∑ T − 1 ∇ θ ln π θ ( a t ∣ s t )
最终，对策略梯度的计算归纳如下：
∇ θ J ( θ ) = ∇ θ E τ ∼ P ( ⋅ ; θ ) [ G ( τ ) ] = ∇ θ ∫ τ P ( τ ; θ ) G ( τ ) d τ Expand expectation = ∫ τ ∇ θ P ( τ ; θ ) G ( τ ) d τ Bring gradient under integral = ∫ τ P ( τ ; θ ) ∇ θ ln P ( τ ; θ ) G ( τ ) d τ Log-derivation trick = E τ ∼ P ( ⋅ ; θ ) [ ∇ θ ln P ( τ ; θ ) G ( τ ) ] Return to expectation form = E τ ∼ P ( ⋅ ; θ ) [ ∑ t = 0 T − 1 ∇ θ ln π θ ( a t ∣ s t ) G ( τ ) ] Grad-log-prob \begin{aligned}
\nabla_{\theta} J(\theta) &= \nabla_{\theta}\mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}[G(\tau)] \\
&= \nabla_{\theta} \int_{\tau} \mathbb{P}(\tau;\theta) G(\tau) d\tau & \text{Expand expectation} \\
&= \int_{\tau} \nabla_{\theta} \mathbb{P}(\tau;\theta) G(\tau) d\tau & \text{Bring gradient under integral} \\
&= \int_{\tau} \mathbb{P}(\tau;\theta) \nabla_{\theta} \ln \mathbb{P}(\tau;\theta) G(\tau) d\tau & \text{Log-derivation trick} \\
&= \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}
\big[\nabla_{\theta} \ln \mathbb{P}(\tau;\theta) G(\tau)\big] & \text{Return to expectation form} \\
&= \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}\Bigg[\sum_{t=0}^{T-1} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t}) G(\tau)\Bigg]& \text{Grad-log-prob}
\end{aligned}
∇ θ J ( θ ) = ∇ θ E τ ∼ P ( ⋅ ; θ ) [ G ( τ )] = ∇ θ ∫ τ P ( τ ; θ ) G ( τ ) d τ = ∫ τ ∇ θ P ( τ ; θ ) G ( τ ) d τ = ∫ τ P ( τ ; θ ) ∇ θ ln P ( τ ; θ ) G ( τ ) d τ = E τ ∼ P ( ⋅ ; θ ) [ ∇ θ ln P ( τ ; θ ) G ( τ ) ] = E τ ∼ P ( ⋅ ; θ ) [ t = 0 ∑ T − 1 ∇ θ ln π θ ( a t ∣ s t ) G ( τ ) ] Expand expectation Bring gradient under integral Log-derivation trick Return to expectation form Grad-log-prob
值得注意的是，这里我们假设 G ( τ ) G(\tau) G ( τ ) θ \theta θ θ \theta θ τ \tau τ G ( τ ) G(\tau) G ( τ ) θ \theta θ τ \tau τ P ( τ ; θ ) \mathbb{P}(\tau;\theta) P ( τ ; θ ) τ \tau τ G ( τ ) G(\tau) G ( τ )
为了突出 G ( τ ) G(\tau) G ( τ ) θ \theta θ ∇ \nabla ∇
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ G ( τ ) ∑ t = 0 T − 1 ∇ θ ln π θ ( a t ∣ s t ) ] \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}\Bigg[G(\tau) \sum_{t=0}^{T-1} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t})\Bigg]
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ G ( τ ) t = 0 ∑ T − 1 ∇ θ ln π θ ( a t ∣ s t ) ]
至此，我们已经分析得到目标函数的梯度。在接下来的一节里，我们将站在深度学习的角度，来粗略给出策略网络的训练步骤。
Training the Policy Network
梯度 ∇ θ J ( θ ) \nabla_{\theta} J(\theta) ∇ θ J ( θ ) { τ n } n = 1 N \{\tau_{n}\}_{n=1}^{N} { τ n } n = 1 N π θ \pi_{\theta} π θ
g ^ = 1 N ∑ n = 1 N G ( τ n ) ∑ t = 0 T n − 1 ∇ θ ln π θ ( a t n ∣ s t n ) \hat{g} = \frac{1}{N} \sum_{n=1}^{N} G(\tau^{n}) \sum_{t=0}^{T_{n}-1} \nabla_{\theta} \ln \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n})
g ^ = N 1 n = 1 ∑ N G ( τ n ) t = 0 ∑ T n − 1 ∇ θ ln π θ ( a t n ∣ s t n )
直观地，每一步参数更新，都需要计算梯度，而梯度需要通过收集 N N N N N N
数据采样 ：重复 N N N n n n
共有 T n T_{n} T n t t t
观测到状态 s t n s_{t}^{n} s t n
从最近更新的策略中随机采样动作 a t n ∼ π θ ( ⋅ ∣ s t n ) a_{t}^{n} \sim \pi_{\theta}(\cdot \mid s_{t}^{n}) a t n ∼ π θ ( ⋅ ∣ s t n )
环境给出下一状态 s t + 1 n s_{t+1}^{n} s t + 1 n r t n = R ( s t , a t , s t + 1 ) r_{t}^{n} = R(s_{t}, a_{t}, s_{t+1}) r t n = R ( s t , a t , s t + 1 )
累计 T n T_{n} T n τ n \tau^{n} τ n
计算轨迹 τ n \tau^{n} τ n G ( τ n ) G(\tau^{n}) G ( τ n )
梯度近似 ：根据 N N N g ^ \hat{g} g ^ ∇ θ J ( θ ) \nabla_{\theta} J(\theta) ∇ θ J ( θ ) 参数更新 ：θ ′ = θ + α ⋅ g ^ \theta^{\prime} = \theta + \alpha \cdot \hat{g} θ ′ = θ + α ⋅ g ^
以上训练步骤仅仅为了直观地了解策略方法的训练过程。而在实际的实现中，mini-batch 的选取可能并非是 N N N
Reward-to-Go Policy Gradient
回顾在 前面章节 得到的策略梯度公式：
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ G ( τ ) ∑ t = 0 T − 1 ∇ θ ln π θ ( a t ∣ s t ) ] \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}\Bigg[G(\tau) \sum_{t=0}^{T-1} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t})\Bigg]
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ G ( τ ) t = 0 ∑ T − 1 ∇ θ ln π θ ( a t ∣ s t ) ]
该公式的核心思路在于，根据获得的奖励之和 G ( τ ) G(\tau) G ( τ ) a t a_{t} a t a t a_{t} a t t t t G t G_{t} G t G ( τ ) G(\tau) G ( τ ) a t a_{t} a t
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ ∑ t = 0 T − 1 G t ∇ θ ln π θ ( a t ∣ s t ) ] \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}\Bigg[\sum_{t=0}^{T-1} G_{t} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t})\Bigg]
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ t = 0 ∑ T − 1 G t ∇ θ ln π θ ( a t ∣ s t ) ]
我们称之为 “reward-to-go” 策略梯度，其中，回报 G t G_{t} G t γ \gamma γ
G t = R t + R t + 1 + ⋯ + R T − 1 = ∑ t ′ = t T − 1 R t ′ G_{t} = R_{t} + R_{t+1} + \cdots + R_{T-1}= \sum_{t^{\prime}=t}^{T-1} R_{t^{\prime}}
G t = R t + R t + 1 + ⋯ + R T − 1 = t ′ = t ∑ T − 1 R t ′
该方法的核心在于，通过 G t G_{t} G t t t t a t a_{t} a t
Baselines in Policy Gradients
baseline 是一个状态 s t s_{t} s t G t G_{t} G t b ( s t ) b(s_{t}) b ( s t )
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ ∑ t = 0 T − 1 ( G t − b ( s t ) ) ∇ θ ln π θ ( a t ∣ s t ) ] \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}\Bigg[\sum_{t=0}^{T-1} \big(G_{t} - b(s_{t})\big) \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t})\Bigg]
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ t = 0 ∑ T − 1 ( G t − b ( s t ) ) ∇ θ ln π θ ( a t ∣ s t ) ]
我们希望 baseline 的引入可以减小方差，同时保持期望不变。对于方差的减小，因具体 baseline 函数而异。经验表明，b ( s t ) = V π ( s t ) b(s_{t}) = V^{\pi}(s_{t}) b ( s t ) = V π ( s t ) b ( s ) b(s) b ( s )
E a ∼ π θ ( ⋅ ∣ s ) [ ∇ θ ln π θ ( a ∣ s ) b ( s ) ] = 0 \operatorname*{\mathbb{E}}_{a \sim \pi_\theta(\cdot \mid s)} [\nabla_{\theta} \ln \pi_\theta(a \mid s) b(s)] = 0
a ∼ π θ ( ⋅ ∣ s ) E [ ∇ θ ln π θ ( a ∣ s ) b ( s )] = 0
那么引入 b ( s ) b(s) b ( s )
EGLP Lemma. 假设随机变量 x x x P ( x ; θ ) \mathbb{P}(x;\theta) P ( x ; θ )
E x ∼ P ( ⋅ ; θ ) [ ∇ θ ln P ( x ; θ ) ] = 0 \operatorname*{\mathbb{E}}_{x \sim \mathbb{P}(\cdot;\theta)} [\nabla_{\theta} \ln \mathbb{P}(x;\theta)] = 0
x ∼ P ( ⋅ ; θ ) E [ ∇ θ ln P ( x ; θ )] = 0
第一步 ，任意概率分布都是归一化的：
∫ x P ( x ; θ ) d x = 1 \int_{x} \mathbb{P}(x;\theta)dx = 1
∫ x P ( x ; θ ) d x = 1
第二步 ，对等式两边求梯度：
∇ θ ∫ x P ( x ; θ ) d x = ∇ θ ⋅ 1 = 0 \nabla_{\theta} \int_{x} \mathbb{P}(x;\theta)dx = \nabla_{\theta} \cdot 1 = 0
∇ θ ∫ x P ( x ; θ ) d x = ∇ θ ⋅ 1 = 0
第三步 ，使用对数导数技巧：
0 = ∇ θ ∫ x P ( x ; θ ) d x = ∫ x ∇ θ P ( x ; θ ) d x = ∫ x P ( x ; θ ) ∇ θ ln P ( x ; θ ) d x = E x ∼ P ( ⋅ ; θ ) [ ∇ θ ln P ( x ; θ ) ] \begin{aligned}
0 &= \nabla_{\theta} \int_{x} \mathbb{P}(x;\theta)dx \\
&= \int_{x} \nabla_{\theta} \mathbb{P}(x;\theta)dx \\
&= \int_{x} \mathbb{P}(x;\theta) \nabla_{\theta} \ln \mathbb{P}(x;\theta)dx \\
&= \operatorname*{\mathbb{E}}_{x \sim \mathbb{P}(\cdot;\theta)} [\nabla_{\theta} \ln \mathbb{P}(x;\theta)]
\end{aligned}
0 = ∇ θ ∫ x P ( x ; θ ) d x = ∫ x ∇ θ P ( x ; θ ) d x = ∫ x P ( x ; θ ) ∇ θ ln P ( x ; θ ) d x = x ∼ P ( ⋅ ; θ ) E [ ∇ θ ln P ( x ; θ )]
EGLP Lemma 证毕。此时，将 π θ ( a ∣ s ) \pi_\theta(a \mid s) π θ ( a ∣ s ) P ( x ; θ ) \mathbb{P}(x;\theta) P ( x ; θ )
E a ∼ π θ ( ⋅ ∣ s ) [ ∇ θ ln π θ ( a ∣ s ) ] = 0 \operatorname*{\mathbb{E}}_{a \sim \pi_\theta(\cdot \mid s)} [\nabla_{\theta} \ln \pi_\theta(a \mid s)] = 0
a ∼ π θ ( ⋅ ∣ s ) E [ ∇ θ ln π θ ( a ∣ s )] = 0
最后，引入 b ( s ) b(s) b ( s )
E a ∼ π θ ( ⋅ ∣ s ) [ ∇ θ ln π θ ( a ∣ s ) b ( s ) ] = 0 \operatorname*{\mathbb{E}}_{a \sim \pi_\theta(\cdot \mid s)} [\nabla_{\theta} \ln \pi_\theta(a \mid s) b(s)] = 0
a ∼ π θ ( ⋅ ∣ s ) E [ ∇ θ ln π θ ( a ∣ s ) b ( s )] = 0
至此，我们证明了引入 baseline 并不会改变期望。一个自然的问题是，除了 G ( τ ) G(\tau) G ( τ ) G t G_{t} G t G t − b ( s t ) G_{t}-b(s_{t}) G t − b ( s t ) a t a_{t} a t
Towards More General Policy Gradient
首先，我们直接给出策略梯度更一般的形式：
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ ∑ t = 0 T − 1 Φ t ∇ θ ln π θ ( a t ∣ s t ) ] \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(\cdot;\theta)}\Bigg[\sum_{t=0}^{T-1} \Phi_{t} \nabla_{\theta} \ln \pi_{\theta}(a_{t} \mid s_{t})\Bigg]
∇ θ J ( θ ) = E τ ∼ P ( ⋅ ; θ ) [ t = 0 ∑ T − 1 Φ t ∇ θ ln π θ ( a t ∣ s t ) ]
前文已经探讨了 Φ t \Phi_{t} Φ t G ( τ ) G(\tau) G ( τ ) G t G_{t} G t G t − b ( s t ) G_{t}-b(s_{t}) G t − b ( s t )
状态价值函数：Φ t = Q π ( s t , a t ) \Phi_{t} = Q^{\pi}(s_{t},a_{t}) Φ t = Q π ( s t , a t )
优势函数：Φ t = A π ( s t , a t ) = Q π ( s t , a t ) − V π ( s t ) \Phi_{t} = A^{\pi}(s_{t},a_{t}) = Q^{\pi}(s_{t},a_{t}) - V^{\pi}(s_{t}) Φ t = A π ( s t , a t ) = Q π ( s t , a t ) − V π ( s t )
对于 Φ t = Q π ( s t , a t ) \Phi_{t} = Q^{\pi}(s_{t},a_{t}) Φ t = Q π ( s t , a t ) Proof for Using Q-Function in Policy Gradient Formula 。而对 Φ t = A π ( s t , a t ) \Phi_{t} = A^{\pi}(s_{t},a_{t}) Φ t = A π ( s t , a t ) Q π ( s t , a t ) Q^{\pi}(s_{t},a_{t}) Q π ( s t , a t ) b ( s t ) = V π ( s t ) b(s_{t}) = V^{\pi}(s_{t}) b ( s t ) = V π ( s t )
后两种形式中的 Q π ( s t , a t ) Q^{\pi}(s_{t},a_{t}) Q π ( s t , a t ) A π ( s t , a t ) A^{\pi}(s_{t},a_{t}) A π ( s t , a t ) π θ \pi_{\theta} π θ θ \theta θ
