While we looked at the surrogate objective in the previous post, we’re actually going to return to the discussion of advantage functions from part 14. In this post, we’ll find a middle ground between full Monte Carlo (MC) returns and one-step bootstrapping in order to estimate the advantage function. This becomes a useful parameter in…

Read more

We are reaching the end of our series, but we still have a few steps to do in our reinforcement learning (RL) journey before we can fully flesh out proximal policy optimization (PPO).  If you’ve been following along and comparing these posts to the Sutton & Barto textbook (or other RL literature), you might have…

Read more

In the previous post, we ended with a straightforward application of the policy gradient in the REINFORCE algorithm, which proves to be a useful stepping stone in our deep reinforcement learning (RL) journey. Here, we updated the parameters of the policy approximator (often a neural network) using this formula: Notice that we are using the…

Read more

In the previous post, we showed how we can substitute our usual ε-greedy policy with a parameterized approximation (often a neural network), we then derived the policy gradient theorem required to optimize this approximator function, and demonstrated how it can be estimated using Monte Carlo sampling. At the very end, we pointed out that the…

Read more

In the previous post, we introduced the breakthrough concept of combining deep learning and reinforcement learning (RL). Instead of recording estimated Q-values in a table, which is intractable for large or continuous state spaces, we approximated those Q-values using a neural network. This simple act spawned the current generation of deep RL, paving the way…

Read more