Databricks and PySpark for Big Data: From Zero to Expert

Course Curriculum

Chapter 1: Introduction to this course

Lecture 1: Course Material

Lecture 2: How to get the most out of the course

Chapter 2: Introduction to Apache Spark and Big Data

Lecture 1: Spark Fundamentals

Lecture 2: How Apache Spark works

Lecture 3: Apache Spark ecosystem and official documentation

Lecture 4: PySpark: cluster management and architecture

Chapter 3: Installation of Spark on premises (Addiotional)

Lecture 1: Spark installation: downloading tools

Lecture 2: Installing Spark: setting environment variables

Lecture 3: Running Spark at the prompt and jupyter notebook

Chapter 4: Spark DataFrames and Apache Spark SQL

Lecture 1: Fundamentals and advantages of DataFrames

Lecture 2: Characteristics of DataFrames and data sources

Lecture 3: Creating DataFrames in PySpark

Lecture 4: Operations with PySpark DataFrames

Lecture 5: Different types of joins in DataFrames

Lecture 6: Consultas SQL en PySpark

Lecture 7: Funciones avanzadas para cargar y exportar datos en PySpark

Chapter 5: Spark Advanced Features

Lecture 1: Advanced Features and Performance Optimization

Lecture 2: BroadCast Join and caching

Lecture 3: User Defined Functions (UDF) and advanced SQL functions

Lecture 4: Handling and imputation of missing values

Chapter 6: Databricks Fundamentals

Lecture 1: Introduction to Databricks

Lecture 2: Databricks Terminology and Databricks Community

Lecture 3: Crear una cuenta gratuita de Databricks

Chapter 7: Databricks Platform

Lecture 1: Introduction to the Databricks environment

Lecture 2: First steps with Databricks

Chapter 8: Databricks Utilities

Lecture 1: Databricks Utilities

Lecture 2: Databricks Utils for managing File System and libraries

Lecture 3: Databricks Utils for notebooks, secrets and Widgets

Chapter 9: ETL, Dataframes and data visualization in Databricks

Lecture 1: Creating and saving DataFrames in Databricks

Lecture 2: Transformation and visualization of data in Databricks

Chapter 10: Machine learning with Databricks and Apache Spark

Lecture 1: Fundamentals of Machine Learning with Spark

Lecture 2: Spark Machine Learning components

Lecture 3: Stages in the development of a Machine Learning model

Lecture 4: Machine Learning Model Definition and Pipeline Development

Lecture 5: Model evaluation with PySpark and Databricks

Lecture 6: Hyperparameter setting and logging in MLFlow

Lecture 7: Predictions with new data and visualization of results

Chapter 11: Databricks Koalas: The Pandas API for Apache Spark

Lecture 1: Spark Koalas Fundamentals

Lecture 2: Feature Engineering with Koalas

Lecture 3: Creating DataFrames with Koalas

Lecture 4: Data Manipulation and DataFrames with Koalas

Lecture 5: Working with missing data in Koalas

Lecture 6: Data visualization and graph generation with Koalas

Lecture 7: Import and export data with Koalas

Chapter 12: Spark Streaming at Databricks

Lecture 1: Spark Streaming Fundamentals

Lecture 2: Example of Streaming word count with Spark Streaming

Lecture 3: Spark Streaming Configurations: Output Modes and Operation Types

Lecture 4: Spark Streaming Capabilities

Lecture 5: Hands-on Lab part I: Spark Streaming in Databricks

Lecture 6: Hands-on Lab part II: Spark Streaming in Databricks

Chapter 13: Real-time forecasting with Databricks, Spark ML and Spark Streaming

Lecture 1: Case Study: Preprocessing Pipeline and ML Model Development

Chapter 14: Delta Lake

Lecture 1: Delta Lake Fundamentals

Lecture 2: Delta Lake features and benefits

Lecture 3: Architecture of a Delta Lake in Azure

Lecture 4: Generate a Delta Lake and query the data

Lecture 5: Unifying Batch and Streamning processes with Delta Lake and ACID transactions

Lecture 6: Preserving data integrity with Schema Enforcement and Evolution in Delta Lake

Lecture 7: Delta Lake version recovery

Lecture 8: DML Consultations at Delta Lake

Lecture 9: Delta Lake performance optimization

Chapter 15: Spark Architecture Concepts

Lecture 1: Spark Optimization Techniques

Lecture 2: Lazy Evaluation

Lecture 3: Wide and Narrow Transformations

Lecture 4: Parquet file in Spark

Lecture 5: Parallelism and Partitions

Lecture 6: Shuffling

Lecture 7: Caching and Storage Levels

Chapter 16: Machine Learning with Databricks and Apache Spark

Lecture 1: Import and exploratory analysis of data

Lecture 2: Variable preprocessing with PySpark and Databricks

Lecture 3: Definition of the Machine Learning model and development of the Pipeline

Lecture 4: Model evaluation with PySpark and Databricks

Lecture 5: Hyperparameter tuning and registration in MLFlow

Lecture 6: Predictions with new data and visualization of the results

Chapter 17: Spark DataFrame API

Lecture 1: Spark SQL and SQL Dataframe API

Lecture 2: Temporary Views vs Global Temporary Views

Lecture 3: Spark Dataframes

Lecture 4: Spark SQL and SQL Dataframe API Lab

Chapter 18: Spark Column Expresions

Lecture 1: Introduction to Spark Column Expresions

Lecture 2: Column Expressions, operators and methods

Lecture 3: DataFrame Transformation Methods

Lecture 4: Subset Rows in Dataframe

Chapter 19: Dataframe Agregations