Orchestration

A pipeline on your laptop is a start. In production, pipelines must run every night, or every week. They must retry when a server hiccups, wait for new data, and send an alert when they fail. Tools that do all this are called orchestrators. Let's see what they offer.

• Run on a schedule
• Retry, wait, and alert
• Many tools, one idea

What an orchestrator does

An orchestrator reads a graph of tasks, and runs it reliably. It starts runs on a schedule, or when an event happens, like new data arriving. It runs tasks in the right order, in parallel when possible. It retries tasks that fail, with delays. It stores the history of every run, with logs. And it sends alerts when something breaks. In short, it is the conductor of your pipeline.

Popular orchestrators

There are many choices. Airflow is the veteran, very widely used, with pipelines written in Python. Prefect and Dagster are modern alternatives, with friendlier developer experience. Kubeflow Pipelines runs each step as a container on Kubernetes, popular for machine learning. Argo Workflows is similar, and Kubernetes native. And cloud services, like those of Amazon, Google, and Azure, offer managed versions. The ideas are the same everywhere, so learn the concepts first.

A weekly retraining DAG

Here is a pipeline written for Airflow. The decorator says, run this every week. Inside, three tasks, prepare data, train the model, and evaluate and register it. The last line connects them, so that evaluation receives the model, which comes from training, which receives the prepared data. Airflow draws this as a graph, runs it every Monday, retries failures, and keeps every run's logs. The task bodies are just your normal Python code.

from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="@weekly", start_date=datetime(2026, 1, 1), catchup=False)
def churn_retraining():

    @task
    def prepare_data():
        ...

    @task
    def train_model(data_path):
        ...

    @task
    def evaluate_and_register(model_path):
        ...

    evaluate_and_register(train_model(prepare_data()))

churn_retraining()

Summary