Reinforcement Learning

How do you teach a dog to sit? You don't show it examples, and you don't explain anything. You wait, and when it happens to sit, you give it a treat. Slowly, it works out what earns a reward. In twenty sixteen, this same idea beat the best Go player in the world.

• How do you teach a dog?
• Rewards, not examples
• The same idea won at Go

The reinforcement loop

There are five words to learn. An agent is the learner. The environment is the world it acts in. A state is the current situation. An action is a choice. And a reward is the score it gets. The agent's goal is to maximise total reward over time, not just right now. That leads to the central dilemma. Should it exploit what it already knows, or explore something new?

Q-learning in a corridor

Here's a real reinforcement learning algorithm, and it's short. Picture a corridor of six squares, with a treasure at the far end. Q is a table of how good each action is, in each square. It starts at zero, so the agent knows nothing. Epsilon decides how often it explores at random. And this line is the famous update. The value of an action moves toward the reward, plus the discounted value of the best next move. Run it. Having started out knowing nothing, it learned to go right in every square. And values grow as it gets closer to the treasure. Nobody told it the answer.

import numpy as np

rng = np.random.default_rng(0)
# a corridor of 6 squares; reaching square 5 pays a reward
Q = np.zeros((6, 2))          # actions: 0 left, 1 right
alpha, gamma, epsilon = 0.5, 0.9, 0.2

for episode in range(200):
    state = 0
    while state != 5:
        explore = rng.random() < epsilon
        action = rng.integers(2) if explore else Q[state].argmax()
        next_state = max(0, state - 1) if action == 0 else state + 1
        reward = 1 if next_state == 5 else 0
        target = reward + gamma * Q[next_state].max()
        Q[state, action] += alpha * (target - Q[state, action])
        state = next_state

print("Best move in each square:", ["LR"[a] for a in Q[:5].argmax(axis=1)])
print("Value of moving right:", Q[:5, 1].round(2))

Best move in each square: ['R', 'R', 'R', 'R', 'R']
Value of moving right: [0.66 0.73 0.81 0.9  1.  ]

Summary