Why Airflow Is Our Go-To Tool for Workflow Orchestration

A Bit of Context...

As data became the most relevant input for companies to generate real business value, Data Engineers had to rise to the occasion. Their challenge: building increasingly complex data pipelines — and keeping them running. That's what workflow orchestration is about, and it's the quiet hero behind every dashboard, ML model, and Gen AI system that works reliably in production.

Data pipelines stand at the core of any data product:

• Ingestion: your data has to come from somewhere, right? From relational databases to APIs and web scraping, Data Engineers ingest from completely different data sources.
• Transformation: some pipelines only clean up ingested data; others transform it into insightful business metrics. Maybe you store it raw and skip this step entirely!
• Storage: the data has to end up somewhere — a file in your data lake, a table in your warehouse or lakehouse. Now it's ready to be consumed by your users, a Business Intelligence tool, or a Machine Learning system.

These pipelines might process batch data (discrete, usually large amounts over a given period) or streaming data (continuous, with no clear separation).

Data pipelines are complex enough, but the devil is in the details. Steps must execute in a specific order and depend on each other: how do you make sure you've ingested from all your sources before transforming? What happens if one source fails?

Say we want to count new users for a given date. We'd build a pipeline that ingests customer data from two databases, transforms it by counting users, and stores the result in a separate table. If those tasks don't run sequentially, the output is simply wrong. Same with failures: miss one source and you're not really counting all your users.

And that's a simple scenario — what if your pipeline has dozens of tasks and dependencies?

Not so long ago, in a galaxy not so far away, data pipelines ran on cron jobs — a simple job scheduler on Unix-like systems. But what was once a simple query joining two tables is now a complex workflow spanning several pipelines, data sources, and systems. You can tell cron isn't enough for modern data orchestration when you find yourself writing custom code to handle:

1. Automatic retries with configurable wait times.
2. Backfilling old data (e.g., you created the workflow on the 10th and want it to run for all dates since the start of the month, then keep running daily).
3. Determining if a job succeeded or failed.
4. Running multiple independent jobs in parallel and evaluating the overall result.
5. Checking that all data dependencies are met before a workflow starts.
6. Monitoring: jobs running, jobs queued, jobs failed, overall workflow status.
7. Re-running part of the workflow and all its dependent tasks.
8. Version-controlling workflow definitions, with procedural logic that varies the workflow by environment or business configuration.

The answer came in the form of workflow orchestration tools. In 2014, looking to solve their own workflow management challenges, Airbnb developed Apache Airflow: a tool that makes executing, troubleshooting, monitoring, and adapting workflows much easier.

Airflow specializes in coordinating tasks that manipulate batches of data. It's not a streaming engine — though Airflow 3 added event-driven scheduling, so workflows can now react to external events instead of only running on a clock.

Airflow, Apache Airflow

Let's go back to the distant past of 2014 for a minute. Your boss asked you to run an ETL job each day at midnight to feed the daily reports.

You created a trusty cron job on a forgotten instance, and that kept your boss happy for a few days. But one morning your boss taps your shoulder (it's 2014, so we're all in open-plan offices instead of our homes!) and angrily tells you the reports are missing yesterday's data — right when they were supposed to present them.

You check the cron job and find nothing. Later you learn that one of the APIs you extract from had a few minutes of downtime at night, and your whole pipeline silently failed.

That's just one of the hidden complexities of workflow orchestration: How do I know how and why it failed? How do I define complex dependencies? Can I retry a few minutes after failure? Can tasks run in parallel? Does it usually fail at the same time of day? How long does it take?

It's hard to ask all that of a cron job. Airflow does it out of the box:

• The order of your tasks is configured in a simple Python file, called a DAG, letting you define actions on both failure and success.
• Retries live in the same DAG file — you decide what happens when a process fails.
• Failures and successes are easy to report to your platform of choice, like email or Slack.
• Best of all: you have the full power of Python to dynamically create and modify any of it.

You only need a handful of concepts, which makes Airflow easy to pick up:

• DAG (Directed Acyclic Graph): a group of tasks with dependencies (a workflow)
• Operator: the type of task to execute (Bash command, Python script, SQL statement, etc.)
• Task: an Operator instance defining one node in the DAG
• DAG Run: an instance in time when a DAG executes
• Task Instance: a specific execution of a task within a DAG Run

What changed since 2022: Airflow 3 landed as the biggest release in the project's history — a modernized UI, DAG versioning (finally!), event-driven scheduling, and first-class support for the ML and AI workloads that dominate today's platforms. The DAGs that used to end in a dashboard now just as often end in a feature store, a model training job, or a Gen AI pipeline. Same orchestration principles, higher stakes: an Airflow data pipeline today is just as likely to feed a model as a report.

Why We Love Airflow at Muttdata

Easy to Use Out of the Box

The entry barrier is low: Airflow's main language is Python, the most widely used language in data teams.

Working with many clients means many environments — some run Airflow in the cloud, others on their own servers. Being able to easily set up and run Airflow anywhere is key to delivering value efficiently.

And the benefit isn't limited to programmers. Anyone with access to Airflow's UI can re-run jobs, check logs, and troubleshoot. The web application surfaces performance, execution times, process status, and retries — making workflow monitoring accessible. Starting, pausing, and ending workflows happens right from the UI.

Team Player

Airflow executes tasks on all the usual providers — Google Cloud, AWS, Microsoft Azure — and instead of hand-coding connections, you can use pre-built Operators to integrate with databases, alerting, and frameworks. That includes the modern lakehouse: our DAGs routinely orchestrate jobs on Databricks, where we're proud to be the LATAM Enterprise AI Partner of the Year.

Scalability & Customization

Airflow scales horizontally as you add tasks and processes — no surprise compute spikes catching you off guard.

But that's not the only reason it scales. Since workflows are code, DAGs can be generated dynamically from a database or configuration file. Adding a DAG for your newest product can require no new code at all, letting anyone schedule powerful pipelines.

Defining workflows as Python also enables higher-level abstractions: functions that generate frequently repeated task combinations, or builder objects that create DAGs from configuration. These characteristics — plus Airflow's popularity — make it the ideal baseline for a DataOps-style platform, which is exactly how we use it.

Based on Python and backed by a huge open-source community, Airflow is easily extensible, with a robust ecosystem of integrations.

Airflow, Managed: Astronomer

If you've heard about Airflow, you may have also heard about Astronomer — a steward of the Apache Airflow project, at the center of its community, releases, and roadmap. We've been Astronomer technology partners for years.

Astronomer offers managed Airflow with commercial support, standing out through seamless deployment, elastic scalability, better system visibility, and reduced day-2 operations. They also maintain excellent learning resources and the Apache Airflow certifications — which we sponsor for our own team.

Getting Started with Airflow

If you're interested in learning more, some links we genuinely recommend:

• Apache Airflow official documentation
• Astronomer Learn guides
• Astronomer certifications

Wrapping Up

We've built data and Machine Learning pipelines with Airflow for ad platforms, real-time bidding systems, marketing campaign optimization, Computer Vision, NLP, and — more recently — the Gen AI systems those pipelines feed. Reliable workflow orchestration is the difference between an AI demo and an AI product, and Airflow remains our go-to for it.

Need help figuring out the best way to boost your business with data? Talk to our team. And if you've got mad dev skills and a data engineering itch, check out our current openings!