Spark Streaming – Stream Processing in Lakehouse – PySpark

Spark Streaming – Stream Processing in Lakehouse – PySpark, available at $74.99, has an average rating of 4.66, with 108 lectures, based on 1342 reviews, and has 14583 subscribers.

You will learn about Real-time Stream Processing Concepts Spark Structured Streaming APIs and Architecture Working with Streaming Sources and Sinks Kafka for Data Engineers Working With Kafka Source and Integrating Spark with Kafka State-less and State-full Streaming Transformations Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Working with Azure Databricks Capstone Project – Streaming application in Lakehouse This course is ideal for individuals who are Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark and Databricks Cloud or Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark and Databricks Cloud It is particularly useful for Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark and Databricks Cloud or Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark and Databricks Cloud.

Enroll now: Spark Streaming – Stream Processing in Lakehouse – PySpark

Summary

Title: Spark Streaming – Stream Processing in Lakehouse – PySpark

Price: $74.99

Average Rating: 4.66

Number of Lectures: 108

Number of Published Lectures: 108

Number of Curriculum Items: 108

Number of Published Curriculum Objects: 108

Original Price: $19.99

Quality Status: approved

Status: Live

What You Will Learn

Real-time Stream Processing Concepts
Spark Structured Streaming APIs and Architecture
Working with Streaming Sources and Sinks
Kafka for Data Engineers
Working With Kafka Source and Integrating Spark with Kafka
State-less and State-full Streaming Transformations
Windowing Aggregates using Spark Stream
Watermarking and State Cleanup
Streaming Joins and Aggregation
Handling Memory Problems with Streaming Joins
Working with Azure Databricks
Capstone Project – Streaming application in Lakehouse

Who Should Attend

Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark and Databricks Cloud
Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark and Databricks Cloud

Target Audiences

Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark and Databricks Cloud
Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark and Databricks Cloud

About the Course

I am creating Apache Spark and Databricks – Stream Processing in Lakehouseusing the Python Language and PySpark API. This coursewill help you understand Real-time Stream processing using Apache Spark and Databricks Cloud and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session-like approach. We will take a live coding approach and explain all the needed concepts.

Capstone Project

This course also includes an End-To-End Capstone project. The project will help you understand the real-life project design, coding, implementation, testing, and CI/CD approach.

Who should take this Course?

I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with those implementing Apache Spark at the ground level.

Spark Version used in the Course.

This Course is using the Apache Spark 3.5. I have tested all the source code and examples used in this Course on Azure Databricks Cloud using Databricks Runtime 14.1.

Course Curriculum

Chapter 1: Before you start

Lecture 1: About the Course

Lecture 2: Course Prerequisite

Lecture 3: Source Code and Other Resources

Lecture 4: Note for Students – Before Start

Chapter 2: Setup your environment

Lecture 1: Spark Development Environments

Lecture 2: Setup your Databricks Community Cloud Environment

Lecture 3: Working in Databricks Workspace

Chapter 3: Getting Started with Spark Streaming

Lecture 1: Batch processing to stream processing

Lecture 2: Your Spark application – Applying Best Practice

Lecture 3: Your first streaming application – Implementing Stream

Lecture 4: Stream Processing Model in Spark

Lecture 5: Create Another Streaming Application

Lecture 6: Stream Triggers

Lecture 7: Incremental Batch Processing

Lecture 8: Streaming Sources and Sinks

Lecture 9: Creating Chain of Streams

Chapter 4: Kafka for Data Engineers

Lecture 1: An Introduction to Kafka

Lecture 2: Creating Kafka Cluster in Cloud

Lecture 3: Kafka Core Concepts

Lecture 4: Producing Data to Kafka Topic

Lecture 5: Consuming Data from Kafka Topic

Lecture 6: Working with Kafka Topic Data

Lecture 7: How to Implement Idempotence

Lecture 8: Working with Kafka Sink

Chapter 5: Streaming Aggregates and State Management

Lecture 1: Streaming Aggregates and State Store

Lecture 2: Incremental Aggregates and Update Mode

Lecture 3: Spark Streaming Output Modes

Lecture 4: Statefull Vs Stateless Aggregation

Lecture 5: Implementing Stateless Streaming Aggregation

Lecture 6: Timebound Stateful Tumbling Window Aggregation

Lecture 7: Watermarking and State Store Cleanup

Lecture 8: Sliding Window Aggregates

Chapter 6: Working with Databricks Platform

Lecture 1: Introduction to Databricks

Lecture 2: Creating Azure Free Account

Lecture 3: Azure Portal Overview

Lecture 4: Creating Azure Databricks Service

Lecture 5: Introduction to Azure Databricks Workspace

Lecture 6: Azure Databricks Architecture

Lecture 7: Creating Azure Databricks Cluster

Lecture 8: Introduction to Databricks Notebooks

Lecture 9: Notebooks Magic Commands and Utilities

Lecture 10: Databricks Notebooks Utilities

Lecture 11: Introduction to Databricks Unity Catalog

Lecture 12: Introduction to Databricks Workflow Jobs

Lecture 13: Introduction to Databricks Rest API

Lecture 14: Introduction to Databricks CLI

Chapter 7: Capstone Project – Implementing Real-time Project in Lakehouse

Lecture 1: Project Scope and Background

Lecture 2: Taking out the operational requirement

Lecture 3: Storage Design

Lecture 4: Implement Data Security

Lecture 5: Implement Resource Policies

Lecture 6: Decouple Data Ingestion

Lecture 7: Design Bronze Layer

Lecture 8: Design Silver and Gold Layer

Lecture 9: Setup your source control

Lecture 10: Setup your environment

Lecture 11: Create a development workspace

Lecture 12: Create and Configure Storage Layer

Lecture 13: Create Unity Catalog Metastore

Lecture 14: Create Catalog and External Locations

Lecture 15: Start Coding

Lecture 16: Test your code

Lecture 17: Load historical data

Lecture 18: Ingest into bronze layer

Lecture 19: Process the silver layer

Lecture 20: Handling multiple updates

Lecture 21: Implementing Gold Layer

Lecture 22: Creating a run script

Lecture 23: Preparing for Integration testing

Lecture 24: Creating Test Data Producer

Lecture 25: Creating Integration Test for Batch mode

Lecture 26: Creating Integration Test for Stream mode

Lecture 27: Implementing CI CD Pipeline

Lecture 28: Develop Build Pipeline

Lecture 29: Develop Release Pipeline

Lecture 30: Creating Databricks CLI Script

Chapter 8: Final Word

Lecture 1: An End is a new Beginning

Chapter 9: Archived – Old Course Content

Lecture 1: Spark Development Environment

Lecture 2: Windows User – Spark Installation Prerequisites

Lecture 3: Windows User – Installing Apache Spark

Lecture 4: Windows User – Setup and test your IDE

Lecture 5: Mac User – Installing Apache Spark

Lecture 6: Mac User – Setup and test your IDE

Lecture 7: Install and run Apache Kafka

Lecture 8: Stream processing model in Spark

Lecture 9: Working with Files and Directories

Lecture 10: Streaming Sources, Sinks and Output Mode

Lecture 11: Fault Tolerance and Restarts

Lecture 12: Introduction to Stream Processing

Lecture 13: Spark Streaming APIs – DStream Vs Structured Streaming

Lecture 14: Creating your first stream processing application

Instructors

Prashant Kumar Pandey
Architect, Author, Consultant, Trainer @ Learning Journal
Learning Journal
Online Training Company

Rating Distribution

1 stars: 6 votes
2 stars: 15 votes
3 stars: 56 votes
4 stars: 393 votes
5 stars: 872 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!

Menu