Mastering Apache Airflow! Deploy to Kubernetes in AWS

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction

Lecture 2: Who this course is for?

Lecture 3: Course objectives

Chapter 2: Apache Airflow overview

Lecture 1: What is Apache Airflow?

Lecture 2: Apache Airflow use cases

Lecture 3: Apache Airflow architecture

Chapter 3: Apache Airflow scheduler

Lecture 1: Introduction

Lecture 2: Sequential Executor

Lecture 3: Local Executor

Lecture 4: Celery Executor

Lecture 5: Kubernetes Executor

Chapter 4: Apache Airflow web server

Lecture 1: Introduction

Lecture 2: DAGs View

Lecture 3: Graph View

Lecture 4: Tree View

Lecture 5: Task Duration View

Lecture 6: Gantt View

Lecture 7: Code view

Chapter 5: Apache Airflow configuration

Lecture 1: Introduction

Lecture 2: Core configuration section

Lecture 3: Scheduler configuration section

Lecture 4: Web server configuration section

Lecture 5: Kubernetes configuration section

Lecture 6: Overriding property in the configuration file

Chapter 6: Apache Airflow components

Lecture 1: DAG

Lecture 2: Plugin

Lecture 3: Operator

Lecture 4: Hook

Lecture 5: Sensor

Lecture 6: XCom

Lecture 7: Variable

Chapter 7: Development environment

Lecture 1: Python

Lecture 2: Docker

Lecture 3: PostgreSQL

Lecture 4: Kubernetes

Lecture 5: Setup project – Working directory

Lecture 6: Setup project – Installing dependencies

Lecture 7: Setup project – Initializing the project

Chapter 8: Create your first simple pipeline

Lecture 1: Introduction

Lecture 2: Create your first DAG

Lecture 3: Create your first plugin

Lecture 4: Create your first operator

Lecture 5: Create your first sensor

Lecture 6: Create your first xcom

Chapter 9: Advanced

Lecture 1: Branching

Lecture 2: Airflow connections

Lecture 3: Airflow logs

Lecture 4: Airflow REST Api

Lecture 5: Airflow metrics

Chapter 10: Apache Airflow in Docker and Docker Compose

Lecture 1: Build Apache Airflow's Docker image

Lecture 2: Run Apache Airflow with Docker Compose

Chapter 11: Apache Airflow in Kubernetes

Lecture 1: Introduction

Lecture 2: What is helm?

Lecture 3: Install helm

Lecture 4: Create a helm chart – PostgreSQL resources

Lecture 5: Create a helm chart – Airflow resources

Lecture 6: Create a helm chart – Security resources

Lecture 7: Deploy to local Kubernetes cluster

Lecture 8: Start using Kubernetes Executor

Chapter 12: Create a DAG executing ETL using Pandas

Lecture 1: Define a DAG

Lecture 2: Define an Operator

Lecture 3: Fetch data from PostgreSQL

Lecture 4: ETL using Pandas

Lecture 5: Make the Operator generic

Lecture 6: Output advices

Chapter 13: Deploy to Kubernetes cluster in AWS

Lecture 1: Install AWS client

Lecture 2: Create IAM user

Lecture 3: AWS permissions

Lecture 4: Create ECR Repository

Lecture 5: Create a Kubernetes cluster in AWS

Lecture 6: Deploy to AWS Kubernetes cluster

Lecture 7: Apache Airflow AWS advices

Chapter 14: Tips to build a real world Apache Airflow application

Lecture 1: Extend Airflow's project structure – part 1

Lecture 2: Extend Airflow's project structure – part 2

Lecture 3: Extend Airflow's project structure – part 3

Lecture 4: DAG with context manager

Lecture 5: Extend default DAG class

Lecture 6: Reuse DAG arguments

Lecture 7: Run webserver and scheduler on different instances