Point-in-Time Correctness

Here is a mistake that makes models look brilliant in testing, and useless in production. It is called data leakage, and one common form is joining features that did not exist yet, at the moment of the prediction. The cure is point in time correctness. Let's see the trap with a tiny, real example.

• Brilliant in testing, useless live
• Features from the future
• Join as of the right moment

How leakage sneaks in

Here is how it happens. You want to train a model to predict churn, as of the tenth of February. You join each customer with their latest known features. But the latest features were recorded in March, after the date you are predicting for. The model quietly learns from the future. In testing, it looks wonderful. In production, those future values do not exist, so the performance collapses. The fix is to join each label, with the feature values that were known at, or before, that moment, and no later.

1. Label at a datepredict as of 10 February
2. Naive joingrabs the latest features
3. Future leaks inMarch values, in a February row
4. Point-in-time joinonly values known by then

Naive join versus point in time

Here is the trap, in a few lines. A table of features, with timestamps, and a table of labels, as of the tenth of February. The naive join takes each customer's latest features. The correct join, merge as of, takes the latest features at or before each label time. Look at customer one. In the naive table, the model sees nine support calls, a value that was only recorded in March. In the point in time table, the model sees two, which is what was known on the tenth of February. Same customer, very different information, and only one of them is legal.

import pandas as pd

features = pd.DataFrame({
    "customer": [1, 1, 1, 2, 2],
    "time": pd.to_datetime(["2026-01-01", "2026-02-01", "2026-03-01", "2026-01-15", "2026-03-15"]),
    "support_calls": [0, 2, 9, 1, 1],
}).sort_values("time")

labels = pd.DataFrame({
    "customer": [1, 2],
    "time": pd.to_datetime(["2026-02-10", "2026-02-10"]),
    "churned": [0, 1],
}).sort_values("time")

latest = features.groupby("customer").last().reset_index()[["customer", "support_calls"]]
naive = labels.merge(latest, on="customer")
correct = pd.merge_asof(labels, features, on="time", by="customer")

print("naive join (uses the future):")
print(naive[["customer", "churned", "support_calls"]].to_string(index=False))
print("point-in-time join:")
print(correct[["customer", "churned", "support_calls"]].to_string(index=False))

naive join (uses the future):
 customer  churned  support_calls
        1        0              9
        2        1              1
point-in-time join:
 customer  churned  support_calls
        1        0              2
        2        1              1

Other kinds of leakage

Leakage has other forms. Target leakage, a feature that is created from the answer, like a field filled in after the customer cancelled. Train test leakage, where the same customer appears in both the training and the test set, or where scaling was fitted on all the data. Time leakage, from random splits of data that is time ordered, instead of splitting by date. The warning sign is always the same. A model that looks too good to be true. When that happens, hunt for leakage first, before celebrating.

Summary