Azure Data Factory | Data Engineering on Azure and Fabric

Course Curriculum

Chapter 1: Course Overview and Set Up

Lecture 1: Course Overview

Lecture 2: Azure Big Data Analytics

Lecture 3: Azure Account Set Up

Lecture 4: Azure UI Overview and Creating Your Subscription

Lecture 5: Services and Regions

Lecture 6: Naming Conventions and Creating the Course Resource Group

Lecture 7: Cost Management and Pricing

Lecture 8: Links and Resources

Chapter 2: Azure Storage

Lecture 1: Azure Storage Solutions Overview

Lecture 2: Classification of Data

Lecture 3: ADF Supported File Formats

Lecture 4: Customer Orders Dataset Download and Overview

Lecture 5: Blueprint for the Data Lake

Lecture 6: Storage Account Creation

Lecture 7: Container Creation and File Upload

Lecture 8: Links and Resources

Chapter 3: ADF – Overview and Set Up

Lecture 1: Azure Data Factory Overview

Lecture 2: ADF Resource Creation

Lecture 3: ADF User Interface and Components Overview

Lecture 4: Links and Resources

Chapter 4: Data Movement

Lecture 1: Copy Activity Overview

Lecture 2: Section Objective

Lecture 3: Copy Activity – Order Items Part 1 (Linked Services and Source/Sink Datasets)

Lecture 4: Copy Activity – Order Items Part 2 (Copy Activity and Pipeline)

Lecture 5: Copy Activity – Customers JSON

Lecture 6: Copy Activity – Orders Parquet

Lecture 7: Copy Activity – Stores and Products JSON

Lecture 8: Organising Data Factory Objects in Folders

Lecture 9: Chaining Activities

Lecture 10: Working with Multiple Files as a Source and Copy Behaviour

Lecture 11: Links and Resources

Chapter 5: Data Transformation

Lecture 1: Data Flow Overview

Lecture 2: Section Objective

Lecture 3: Data Flow Demo and User Interface

Lecture 4: Data Flow Debug (Pricing)

Lecture 5: Data Flow Debug (Row Limit)

Lecture 6: Source and Sink Partitioning

Lecture 7: Select Transformation

Lecture 8: Cast Transformation

Lecture 9: Import Projection

Lecture 10: Derived Column Transformation

Lecture 11: Raw to Cleansed Data Flow Requirement

Lecture 12: Raw to Cleansed Data Flow Walkthrough

Lecture 13: Partitioned Datasets

Lecture 14: Filter and Sort Transformations

Lecture 15: Aggregate Transformation

Lecture 16: Join Transformation

Lecture 17: Conditional Split and Union Transformations

Lecture 18: Cleansed to Structured Data Flow Requirement

Lecture 19: Cleansed to Structured Data Flow Walkthrough

Lecture 20: Structured to Analytics Data Flow Requirement

Lecture 21: Structured to Analytics Data Flow Walkthrough

Lecture 22: Links and Resources

Chapter 6: Scheduling and Chaining Pipelines for Execution

Lecture 1: Execute Pipeline Activity

Lecture 2: Data Flow Debug Session vs Activity Runtime

Lecture 3: Schedule Triggers

Lecture 4: Storage Events Trigger

Lecture 5: Tumbling Window Triggers

Lecture 6: Links and Resources

Chapter 7: Control Flow, Parameters and Variables

Lecture 1: Data Lake Preparation

Lecture 2: Delete Activity

Lecture 3: Get Metadata Activity

Lecture 4: Lookup Activity

Lecture 5: If Condition and Switch Activities

Lecture 6: Introduction to Parameters in ADF

Lecture 7: Introduction to Variables in ADF

Lecture 8: ForEach Activity

Lecture 9: Metadata Based Approach (Example)

Lecture 10: Course Project Activity: Dynamic Datasets and Folder Structure

Lecture 11: Links and Resources

Chapter 8: Azure SQL Database for Structured Data

Lecture 1: Section Overview

Lecture 2: Creating our Azure SQL Resource

Lecture 3: Creating our Schema and Tables

Lecture 4: Adding a Unique Identifier Column to Structured Orders Data

Lecture 5: Creating Linked Service and Datasets for SQL

Lecture 6: Important Note Regarding the Dynamic Datasets

Lecture 7: Copy Activity with SQL Sink

Lecture 8: Data Flow with SQL Sink

Lecture 9: Course Project Activity: Storing our Structured Data in the SQL Database

Lecture 10: Links and Resources

Chapter 9: Tumbling Window Triggers

Lecture 1: Tumbling Window Triggers Overview

Lecture 2: Ingest Data for Specific Time Intervals

Lecture 3: Tumbling Window Trigger Dependencies

Lecture 4: Links and Resources

Chapter 10: Authentication and Access

Lecture 1: Overview

Lecture 2: Azure Key Vault

Lecture 3: System Assigned Managed Identities

Lecture 4: User Assigned Managed Identities

Lecture 5: Service Principals

Lecture 6: Fine Grained Access Control for Storage Accounts