Snowflake cloud database with ELT(Airflow+Python+Talend)

Course Curriculum

Chapter 1: /***** Module 1 *****/

Lecture 1: Module 1 Intro

Chapter 2: prerequisites

Lecture 1: Setup snowflake environment

Lecture 2: Virtual machine download

Lecture 3: Mount local folder to vm and start vm

Lecture 4: Setup Vs code for VM

Lecture 5: Configure connection details

Lecture 6: Github link

Chapter 3: Airflow intro

Lecture 1: Airflow introduction

Lecture 2: Airflow architecture

Lecture 3: First Dag

Lecture 4: Introduction to metadata DB

Lecture 5: Airflow scheduler

Lecture 6: Scheduling in airflow

Lecture 7: Executors in airflow

Lecture 8: Airflow webserver

Lecture 9: Handle failures

Chapter 4: Snowflake airflow ELT

Lecture 1: Introduction

Lecture 2: Load to snowflake example 1

Lecture 3: [Demo] Load to snowflake example 1

Lecture 4: Load to snowflake example 2

Lecture 5: Load to snowflake example 3

Chapter 5: Snowflake python api intro — Basics

Lecture 1: Introduction

Lecture 2: Connection test 1

Lecture 3: Connection test 2

Lecture 4: execute method

Lecture 5: cursor attributes

Lecture 6: execute many

Lecture 7: execute async part 1

Lecture 8: execute async part 2

Lecture 9: execute async part 3

Lecture 10: Formatting: pyformat

Lecture 11: Formatting: qmark

Lecture 12: Result metadata

Lecture 13: Type code

Chapter 6: Snowflake python api intro — Advanced

Lecture 1: Fetch pandas arrow

Lecture 2: Context manager

Lecture 3: Result batch

Lecture 4: Get result batch

Lecture 5: Fetch arrow batches

Lecture 6: Fetch pandas batches

Lecture 7: To pandas arrow

Lecture 8: Execute stream

Lecture 9: Execute string

Lecture 10: Cursor dict

Chapter 7: Snowflake python ELT

Lecture 1: Copy data : s3 to snowflake

Lecture 2: Copy data : s3 to snowflake ( parallel )

Lecture 3: Execute ELT query : Python way

Lecture 4: Schedule airflow job : For python code

Chapter 8: Talend introduction.

Lecture 1: Talend as open source ETL tool.

Lecture 2: Download and install talend.

Lecture 3: Talend workspace introduction.

Lecture 4: Build first job and configure talend

Lecture 5: Build sample job : Level 1

Lecture 6: Build sample job : Level 2

Lecture 7: Build sample job : Level 3

Lecture 8: Building sample job : Level4

Lecture 9: Build sample job : Level5

Lecture 10: Build sample job : Schedule using airflow

Lecture 11: Summary

Chapter 9: / **** Module 2 **** /

Lecture 1: Module 2 intro

Chapter 10: Create generic components.

Lecture 1: Analyst requirement.

Lecture 2: S3 to snowflake copy app demo

Lecture 3: Requirement to process data.

Lecture 4: Elt app demo

Lecture 5: Scheduling job.

Lecture 6: Writing generic dag code part1

Lecture 7: Writing generic dag code part2

Lecture 8: Writing generic dag code part3

Chapter 11: Building workflow

Lecture 1: Introduction

Lecture 2: High level design

Lecture 3: Copy data from s3

Lecture 4: Parsing data

Lecture 5: Audit src records

Lecture 6: Archive and create view part 1

Lecture 7: Archive and create view part 2

Lecture 8: Score records part 1

Lecture 9: Score records part 2

Lecture 10: prepare dag code

Lecture 11: Execute airflow job

Lecture 12: Improve the approach

Lecture 13: Parameter registry

Lecture 14: Summary

Chapter 12: Nyc traffic data demo

Lecture 1: Introduction.

Lecture 2: Nyc traffic data overview

Lecture 3: Load sample data

Lecture 4: Nyc full data load

Lecture 5: Snow sight Dashboard

Chapter 13: /****Bonus sections starts****/