Deep Reinforcement Learning
Learning by doing.
In twenty thirteen, a small London company showed a network learning to play old Atari games, from nothing but pixels and the score. Google bought the company. Three years later, its descendant beat the world champion at Go, a game that experts thought was decades away. How do you train a network with no labelled answers?
- Atari from pixels alone
- Then the Go champion
- No labelled answers
Deep Q-learning
Standard Q learning uses a table, with one value for each state and action. But a game has far too many states. So we use a neural network to estimate the values. Two tricks made it stable. A replay memory stores past experiences, and it trains on random samples, which breaks harmful correlations. And a target network keeps the goal steady. That was the system called D Q N.
A tiny DQN
Here's a miniature D Q N. The world is a simple line, with a reward at the far right end. The agent acts, stores what happened in its memory, and then trains on a random batch from that memory. The target is the reward, plus the discounted best value of the next state. Run it. It has learned to go right, from any position, and nobody told it to. It found out by trying, failing, and remembering.
import numpy as np
import torch
from torch import nn
class Line:
def reset(self):
self.x = 0.0
return torch.tensor([self.x])
def step(self, a):
self.x += 0.2 if a == 1 else -0.2
done = abs(self.x) >= 1.0
return torch.tensor([self.x]), 1.0 if self.x >= 1.0 else 0.0, done
torch.manual_seed(0)
rng = np.random.default_rng(0)
net = nn.Sequential(nn.Linear(1, 32), nn.ReLU(), nn.Linear(32, 2))
opt = torch.optim.Adam(net.parameters(), lr=0.01)
env, memory = Line(), []
for episode in range(150):
s, done, n = env.reset(), False, 0
while not done and n < 40:
explore = rng.random() < max(0.05, 1 - episode / 100)
a = int(rng.integers(2)) if explore else net(s).argmax().item()
s2, r, done = env.step(a)
memory.append((s, a, r, s2, done)); s = s2; n += 1
batch = [memory[i] for i in rng.integers(0, len(memory), 16)]
bs, ba, br, bs2, bd = zip(*batch)
bs, bs2 = torch.stack(bs), torch.stack(bs2)
with torch.no_grad():
best = net(bs2).max(1).values
target = torch.tensor(br) + 0.95 * best * (1 - torch.tensor(bd).float())
q = net(bs).gather(1, torch.tensor(ba).unsqueeze(1)).ravel()
loss = ((q - target) ** 2).mean()
opt.zero_grad(); loss.backward(); opt.step()
for x in [-0.6, 0.0, 0.6]:
print(f"at {x:+.1f} go", "RIGHT" if net(torch.tensor([x])).argmax() == 1 else "LEFT")at -0.6 go RIGHT at +0.0 go RIGHT at +0.6 go RIGHT
Policy gradients
There's a second approach. Instead of estimating values, directly learn which action to choose. The rule is very simple. If an action paid well, make it more likely. That's the policy gradient, and the core of many modern methods, like P P O. Here, three machines have different payouts. Run it. The network shifts almost all of its chances to the best machine. It's also how language models were fine tuned using human feedback, by making well-rated answers more likely.
import torch
from torch import nn
torch.manual_seed(0)
policy = nn.Sequential(nn.Linear(1, 16), nn.ReLU(), nn.Linear(16, 3))
opt = torch.optim.Adam(policy.parameters(), lr=0.02)
state = torch.tensor([[1.0]])
payouts = torch.tensor([0.1, 0.9, 0.3])
for step in range(300):
dist = torch.distributions.Categorical(logits=policy(state))
action = dist.sample()
reward = payouts[action]
loss = (-dist.log_prob(action) * reward).mean()
opt.zero_grad(); loss.backward(); opt.step()
print("Chances:", policy(state).softmax(1)[0].round(decimals=2).tolist())Chances: [0.0, 1.0, 0.0]
Summary
Value methods learn how good each action is, and policy methods learn what to do. Both learn from rewards. Now, you've got every major family. But a powerful model that nobody understands can be dangerous. Next, how to see inside a network.
Key takeaways- DQN: a network estimates values
- Replay memory stabilises it
- Policy gradients favour what paid off