AI Hints
Home›Statistics & Math for ML Tutorial›Loss and Logistic Regression
Statistics & Math for ML · Lesson 104 of 138

The Sigmoid

Squash into 0 to 1.

⏱ 2 min read▶ Video at 2:03:43🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A line can output any number, like minus five, or a hundred. But a probability must be between zero and one. The sigmoid function squashes any number into that range. It's the bridge from a line to a probability.

The Sigmoid
From the video: The Sigmoid
  • Lines output any number
  • Probabilities must be 0 to 1
  • The sigmoid squashes

The S curve

The sigmoid is one, divided by one plus E to the minus Z. Big positive inputs give nearly one. Big negative inputs give nearly zero. And zero gives exactly a half. Run it. It makes a smooth S shaped curve.

sigmoid.pypython
import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-8, 8, 200)
plt.plot(z, sigmoid(z))
plt.axhline(0.5, color="#22d3ee", ls="--"); plt.axvline(0, color="#22d3ee", ls="--")
plt.title("sigmoid(z) = 1 / (1 + e^-z)")
plt.savefig("sigmoid.png")

A handy derivative

The sigmoid has a beautifully simple derivative. Sigmoid of Z, times one minus sigmoid of Z. We used it in the backpropagation lesson. Its slope is biggest at zero, and nearly flat at the ends. When outputs get stuck near zero or one, the gradients get tiny, which can slow learning.

keyvalue
σ(z)1 ÷ (1 + e⁻ᶻ)
σ(0)0.5
σ'(z)σ(z) × (1 − σ(z))
Flat endstiny gradients

Summary

The sigmoid squashes any number into a probability between zero and one. Next, using it to build a classifier, logistic regression.

Key takeaways
  • 1 / (1 + e^−z)
  • 0.5 at zero
  • S-shaped
Next up: Logistic Regression — probabilities from a line