The Complete Hands-On Introduction to Apache Airflow

Course Curriculum

Chapter 1: Course Introduction

Lecture 1: Prerequisites

Lecture 2: Course Objectives

Lecture 3: Who I am?

Lecture 4: Development Environment

Chapter 2: Getting Started with Airflow

Lecture 1: Why Airflow?

Lecture 2: What is Airflow?

Lecture 3: Core Components

Lecture 4: Core Concepts

Lecture 5: Airflow is not…

Lecture 6: The Different Architectures

Lecture 7: How does it work?

Lecture 8: [Practice] Installing Apache Airflow

Lecture 9: What is Docker?

Lecture 10: The docker-compose file

Lecture 11: Key Takeaways

Chapter 3: The important views of the Airflow UI

Lecture 1: The DAGs View

Lecture 2: The Grid View

Lecture 3: The Graph View

Lecture 4: The Landing Times View

Lecture 5: The Calendar View

Lecture 6: The Gantt View

Lecture 7: The Code View

Lecture 8: Wrap up!

Chapter 4: Coding Your First Data Pipeline with Airflow

Lecture 1: The Project

Lecture 2: Advices

Lecture 3: What is a DAG?

Lecture 4: DAG Skeleton

Lecture 5: What is an Operator?

Lecture 6: Providers

Lecture 7: Create a Table

Lecture 8: Create a connection

Lecture 9: The secret weapon!

Lecture 10: What is a Sensor?

Lecture 11: Is the API available?

Lecture 12: Create the connection user_api

Lecture 13: Extract users

Lecture 14: Process users

Lecture 15: Before running process_user

Lecture 16: What is a Hook?

Lecture 17: Store users

Lecture 18: Order matters!

Lecture 19: Your DAG in action!

Lecture 20: DAG Scheduling

Lecture 21: Backfilling: How does it work?

Lecture 22: Wrap up!

Chapter 5: The New Way of Scheduling DAGs

Lecture 1: Why do you need that feature?

Lecture 2: What is a Dataset?

Lecture 3: Adios schedule_interval!

Lecture 4: Create the Producer DAG

Lecture 5: Create the Consumer DAG

Lecture 6: Track your Datasets with the new view!

Lecture 7: Wait for many datasets

Lecture 8: Dataset limitations

Chapter 6: Databases and Executors

Lecture 1: What's an executor?

Lecture 2: The default config

Lecture 3: The Sequential Executor

Lecture 4: The Local Executor

Lecture 5: The Celery Executor

Lecture 6: The current config

Lecture 7: Add the DAG parallel_dag.py into the dags folder

Lecture 8: Monitor your tasks with Flower

Lecture 9: Remove DAG examples

Lecture 10: Running tasks on Celery Workers

Lecture 11: What is a queue?

Lecture 12: Add a new Celery Worker

Lecture 13: Create a queue to better distribute tasks

Lecture 14: Send a task to a specific queue

Lecture 15: Concurrency, the parameters you must know!

Chapter 7: Implementing Advanced Concepts in Airflow

Lecture 1: Adios repetitive patterns

Lecture 2: Add the DAG group_dag.py

Lecture 3: How to use SubDAGs?

Lecture 4: Adios SubDAGs, welcome TaskGroups!

Lecture 5: Add the DAG xcom_dag.py

Lecture 6: Sharing data between tasks with XComs

Lecture 7: [Practice] XComs in action!

Lecture 8: Choosing a specific path in your DAG

Lecture 9: [Practice] Executing a task according to a condition

Lecture 10: Trigger rules or how tasks get triggered