We are reaching the end of our series, but we still have a few steps to do in our reinforcement learning (RL) journey before we can fully flesh out proximal policy optimization (PPO).  If you’ve been following along and comparing these posts to the Sutton & Barto textbook (or other RL literature), you might have…

Read more

In the previous post, we ended with a straightforward application of the policy gradient in the REINFORCE algorithm, which proves to be a useful stepping stone in our deep reinforcement learning (RL) journey. Here, we updated the parameters of the policy approximator (often a neural network) using this formula: Notice that we are using the…

Read more

In the previous post, we showed how we can substitute our usual ε-greedy policy with a parameterized approximation (often a neural network), we then derived the policy gradient theorem required to optimize this approximator function, and demonstrated how it can be estimated using Monte Carlo sampling. At the very end, we pointed out that the…

Read more

In the previous post, we introduced the breakthrough concept of combining deep learning and reinforcement learning (RL). Instead of recording estimated Q-values in a table, which is intractable for large or continuous state spaces, we approximated those Q-values using a neural network. This simple act spawned the current generation of deep RL, paving the way…

Read more

Previously, we looked at how Q-learning used off-policy temporal difference (TD) updates to converge on an optimal policy. This reinforcement learning (RL) algorithm works surprisingly well, but it requires the environment to have relatively small, discrete state and action spaces. The Q-table can quickly grow to intractable sizes with environments that have a large number…

Read more