-
Reinforcement Learning Part 10: Q-Learning
One of the biggest breakthroughs in reinforcement learning (RL) occurred in 1989 with Chris Watkins’s paper, Learning from Delayed Rewards. In it, he proposed Q-learning, which decouples the experience gathered from the policy update. In other words, the agent can collect experience using one policy (called the behavior policy) while updating a different policy (called…
-
Reinforcement Learning Part 9: TD(λ) and Eligibility Traces
In the previous post, we saw how temporal difference (TD) learning updated value predictions in the middle of an episode rather than waiting to the very end, like we do with Monte Carlo (MC) methods. If you recall, the TD(0) algorithm updates value estimates after every step using a single reward in order to bootstrap…
-
Reinforcement Learning Part 8: Temporal-Difference (TD) Learning
Temporal Difference (TD) learning is one of the foundational concepts in reinforcement learning (RL). It combines the notion of updating estimates before the final outcome is known, similar to how dynamic programming (DP) works, with the notion of learning directly from experience, like we saw with the Monte Carlo (MC) methods in part 7. MC…
-
Reinforcement Learning Part 7: Monte Carlo Methods
In the previous post, we saw how dynamic programming (DP) could be used to solve the Bellman equations, but they required knowledge of the environment’s transition probabilities. Unfortunately, we do not have that luxury in most real-world reinforcement learning (RL) problems. In DP, we have a model of the environment, which means that the transition…
-
Reinforcement Learning Part 6: Dynamic Programming
Over the past few posts, we built up to the Bellman optimality equations, which allows us to express the existence of an optimal policy that would maximize total rewards from a given environment, assuming the environment qualifies as a Markov Decision Process (MDP). Going forward, we will focus on algorithms that attempt to find policies…
