Databricks – Master Azure Databricks for Data Engineers

Course Curriculum

Chapter 1: Before you start

Lecture 1: Course Prerequisites

Lecture 2: About the Course

Lecture 3: How to access Course Material and Resources

Lecture 4: Note for Students – Before Start

Chapter 2: Introduction

Lecture 1: Introduction to Data Engineering

Lecture 2: Apache Spark to Data Engineering Platform

Lecture 3: Introduction to Databricks Platform

Chapter 3: Getting Started

Lecture 1: What will you learn in this section

Lecture 2: Creating Azure Cloud Account

Lecture 3: Azure Portal Overview

Lecture 4: Creating Databricks Workspace Service

Lecture 5: Introduction to Databricks Workspace

Lecture 6: Azure Databricks Platform Architecture

Chapter 4: Working in Databricks Workspace

Lecture 1: What will you learn in this section

Lecture 2: How to create Spark Cluster

Lecture 3: Working with Databricks Notebook

Lecture 4: Notebook Magic Commands

Lecture 5: Databricks Utilities Package

Chapter 5: Working with Databricks File System – DBFS

Lecture 1: What will you learn in this section

Lecture 2: Introduction to DBFS

Lecture 3: Working with DBFS Root

Lecture 4: Mounting ADLS to DBFS

Chapter 6: Working with Unity Catalog

Lecture 1: What will you learn in this section

Lecture 2: Introduction to Unity Catalog

Lecture 3: Setup Unity Catalog

Lecture 4: Unity Catalog User Provisioning

Lecture 5: Working with Securable Objects

Chapter 7: Working with Delta Lake and Delta Tables

Lecture 1: What will you learn in this section

Lecture 2: Introduction to Delta Lake

Lecture 3: Creating Delta Table

Lecture 4: Sharing data for External Delta Table

Lecture 5: Reading Delta Table

Lecture 6: Delta Table Operations

Lecture 7: Delta Table Time Travel

Lecture 8: Convert Parquet to Delta

Lecture 9: Delta Table Schema Validation

Lecture 10: Delta Table Schema Evolution

Lecture 11: Look Inside Delta Table

Lecture 12: Delta Table Utilities and Optimization

Chapter 8: Working with Databricks Incremental Ingestion Tools

Lecture 1: What will you learn in this section

Lecture 2: Architecture and Need for Incremental Ingestion

Lecture 3: Using Copy Into with Manual Schema Evolution

Lecture 4: Using Copy Into with Automatic Schema Evolution

Lecture 5: Streaming Ingestion with Manual Schema Evolution

Lecture 6: Streaming Ingestion with Automatic Schema Evolution

Lecture 7: Introduction to Databricks Autoloader

Lecture 8: Autoloader with Automatic Schema Evolution

Chapter 9: Working with Databricks Delta Live Tables (DLT)

Lecture 1: What will you learn in this section

Lecture 2: Introduction to Databricks DLT

Lecture 3: Understand DLT Use Case Scenario

Lecture 4: Setup DLT Scenario Dataset

Lecture 5: Creating DLT Workload in SQL

Lecture 6: Creating DLT Pipeline for your Workload

Lecture 7: Creating DLT Workload in Python

Chapter 10: Databricks Project and Automation Features

Lecture 1: What will you learn in this section

Lecture 2: Working with Databricks Repos

Lecture 3: Working with Databricks Workflows

Lecture 4: Working with Databricks Rest API

Lecture 5: Working with Databricks CLI

Chapter 11: Capstone Project

Lecture 1: Project Scope and Background

Lecture 2: Taking out the operational requirement

Lecture 3: Storage Design

Lecture 4: Implement Data Security

Lecture 5: Implement Resource Policies

Lecture 6: Decouple Data Ingestion

Lecture 7: Design Bronze Layer

Lecture 8: Design Silver and Gold Layer

Lecture 9: Setup your environment

Lecture 10: Create a workspace

Lecture 11: Create and Storage Layer

Lecture 12: Setup Unity Catalog

Lecture 13: Create Metadata Catalog and External Locations

Lecture 14: Setup your source control

Lecture 15: Start Coding

Lecture 16: Test your code

Lecture 17: Load historical data

Lecture 18: Ingest into bronze layer

Lecture 19: Process the silver layer

Lecture 20: Handling multiple updates

Lecture 21: Implementing Gold Layer

Lecture 22: Creating a run script

Lecture 23: Preparing for Integration testing

Lecture 24: Creating Test Data Producer

Lecture 25: Creating Integration Test for Batch mode

Lecture 26: Creating Integration Test for Stream mode

Lecture 27: Implementing CI CD Pipeline

Lecture 28: Develop Build Pipeline

Lecture 29: Develop Release Pipeline

Lecture 30: Creating Databricks CLI Script