Build End – End Azure Data Engineering Project

Course Curriculum

Chapter 1: Let's understand Azure Resources and How to create them

Lecture 1: Architecture

Lecture 2: Introduction of ADF

Lecture 3: Introduction of Blob Storage

Lecture 4: Introduction of AzureDataLake Gen2

Lecture 5: Introduction of Key Vault

Lecture 6: What is SPN and How to register SPN?

Lecture 7: Introduction of SQL

Lecture 8: Introduction of Azure Databricks

Lecture 9: Introduction of Azure Logic App

Lecture 10: Introduction of Azure Automation Account

Lecture 11: What is API? and How to Register API_Key ?

Chapter 2: Data Preparation and Uploading into the respective sources

Lecture 1: Import files,data into Azure Blob, DataLake Gen2, SQL Database

Chapter 3: Ingest Data from multiple source

Lecture 1: Create LinkedServices and Grant Access to Resource

Lecture 2: Create Datasets and Ingest CSV files

Lecture 3: Copy SQL tables from Server to DataLake Gen2 using parameterization

Lecture 4: Unzip files and Ingest them into Data Lake Gen2 with single pipeline

Lecture 5: Fetch Data from API using Rest API and Web Activity to fetch credentials

Lecture 6: Setup Repository and commit ADF changes to Github Repository

Lecture 7: Updates on ADF Pipeline

Lecture 8: Create Mount Point of Azure Blob Storage using Databricks

Lecture 9: Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks

Lecture 10: Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake

Chapter 4: Cleaning Process and Data Quality Check

Lecture 1: Clean the Data using AUTOLOADER in Databricks – Part 1

Lecture 2: Clean the Data using AUTOLOADER in Databricks – Part 2

Lecture 3: Clean the Data using AUTOLOADER in Databricks – Part 3

Lecture 4: Clean the Data using Batch processing in Databricks

Lecture 5: Data Quality checks on Cleansed Layer using Delta

Lecture 6: Commit Databricks Notebook using Repo feature in GitHub

Chapter 5: Create Dimesions and Fact tables

Lecture 1: Create Dimesions and Fact tables

Chapter 6: Publish the data into SQL DataBase using ADF

Lecture 1: Publish final tables into SQL Databases using Azure Databricks Delta Lake

Chapter 7: Setup complete ADF Pipelines

Lecture 1: Create Master Pipeline to run the complete project

Chapter 8: Send Data to End user using Logic App

Lecture 1: Multiple Ways to Send Data files to End Users

Lecture 2: Optimize way to send Data Files to End Users

Chapter 9: Run Automation Account script using Runbook

Lecture 1: Switch to SQL DW and Start/Stop the SQL DW using Automation Account

Chapter 10: Data Quality Layer between Mart and Publish

Lecture 1: Build Data Checks Utility to check Mart logics in Databricks

Chapter 11: CI-CD using Azure Devops and Github Actions

Lecture 1: Azure Data Factory Deployment

Lecture 2: Azure Blob Storage and GEN2 files Deployment

Lecture 3: Azure Databricks Deployment using Github Action

Lecture 4: Add Trigger and Global Parameters and Fix CI-CD for Azure Data Factory

Chapter 12: Handle corner cases in ADF

Lecture 1: How to use Filter Activity to handle Failure Runs in ADF

Chapter 13: Build Dashboard and Report using PowerBI

Lecture 1: Import Data and Create Report