Mastering Databricks & Apache spark -Build ETL data pipeline

Mastering Databricks & Apache spark -Build ETL data pipeline, available at $54.99, has an average rating of 4.3, with 47 lectures, based on 419 reviews, and has 2446 subscribers.

You will learn about Databricks Build your first data pipeline to process CSV, JSON, XML Orchestrate data pipeline on Azure data factory Spin up spark cluster Delta tables Concept of time travel and vacuum on delta tables Apache Spark SQL Filtering Dataframe Renaming, drop, Select, Cast Aggregation operations SUM, AVERAGE, MAX, MIN Rank, Row Number, Dense Rank Building dashboards Build Complete project Build End to End data pipeline This course is ideal for individuals who are Data engineer or People who are interested in build End to End ETL data pipeline or Learn fundamentals commands in Python, Apache Spark SQL, Scala It is particularly useful for Data engineer or People who are interested in build End to End ETL data pipeline or Learn fundamentals commands in Python, Apache Spark SQL, Scala.

Enroll now: Mastering Databricks & Apache spark -Build ETL data pipeline

Summary

Title: Mastering Databricks & Apache spark -Build ETL data pipeline

Price: $54.99

Average Rating: 4.3

Number of Lectures: 47

Number of Published Lectures: 47

Number of Curriculum Items: 47

Number of Published Curriculum Objects: 47

Original Price: $22.99

Quality Status: approved

Status: Live

What You Will Learn

Databricks
Build your first data pipeline to process CSV, JSON, XML
Orchestrate data pipeline on Azure data factory
Spin up spark cluster
Delta tables
Concept of time travel and vacuum on delta tables
Apache Spark SQL
Filtering Dataframe
Renaming, drop, Select, Cast
Aggregation operations SUM, AVERAGE, MAX, MIN
Rank, Row Number, Dense Rank
Building dashboards
Build Complete project
Build End to End data pipeline

Who Should Attend

Data engineer
People who are interested in build End to End ETL data pipeline
Learn fundamentals commands in Python, Apache Spark SQL, Scala

Target Audiences

Data engineer
People who are interested in build End to End ETL data pipeline
Learn fundamentals commands in Python, Apache Spark SQL, Scala

Welcome to the course on Mastering Databricks & Apache spark -Build ETL data pipeline

Databricks combines the best of data warehouses and data lakes into a lakehouse architecture. In this course we will be learning how to perform various operations in Scala, Python and Spark SQL. This will help every student in building solutions which will create value and mindset to build batch process in any of the language. This course will help in writing same commands in different language and based on your client needs we can adopt and deliver world class solution. We will be building end to end solution in azure databricks.

Key Learning Points

We will be building our own cluster which will process our data and with one click operation we will load different sources data to Azure SQL and Delta tables
After that we will be leveraging databricks notebook to prepare dashboard to answer business questions
Based on the needs we will be deploying infrastructure on Azure cloud
These scenarios will give student 360 degree exposure on cloud platform and how to step up various resources
All activities are performed in Azure Databricks

Fundamentals

Databricks
Delta tables
Concept of versions and vacuum on delta tables
Apache Spark SQL
Filtering Dataframe
Renaming, drop, Select, Cast
Aggregation operations SUM, AVERAGE, MAX, MIN
Rank, Row Number, Dense Rank
Building dashboards
Analytics

This course is suitable for Data engineers, BI architect, Data Analyst, ETL developer, BI Manager

Course Curriculum

Chapter 1: Getting Started with Databricks

Lecture 1: Introduction

Lecture 2: What is Databricks

Lecture 3: Project

Lecture 4: Create Azure Account

Lecture 5: Setting up databricks environment

Lecture 6: Importing Notebooks

Lecture 7: Understanding Distributed Processing

Lecture 8: How to create cluster

Lecture 9: Notebook

Lecture 10: Why Databricks

Lecture 11: Create table or dataframe by uploading data

Chapter 2: Extraction of Data

Lecture 1: Understanding ETL

Lecture 2: Extraction of data from Azure account

Lecture 3: Adding Schema to data files

Lecture 4: Unmanaged tables

Lecture 5: Managed tables

Chapter 3: Transformation of Data

Lecture 1: Window Functions

Lecture 2: Scala – Filtering Dataframe

Lecture 3: Scala – Common Operations

Lecture 4: Scala – Aggregation commands

Lecture 5: Scala – Rank, Row Number, Dense Rank

Lecture 6: Python – Filtering Dataframe

Lecture 7: Python – Common Operations

Lecture 8: Python – Aggregation commands

Lecture 9: Python – Rank, Row Number, Dense Rank

Lecture 10: Spark SQL – Common Operations

Lecture 11: Spark SQL – Aggregation Commands

Lecture 12: Spark SQL – Rank, Row Number, Dense Rank

Lecture 13: Spark SQL – Global View

Lecture 14: Spark SQL – Temp View

Lecture 15: Joins

Lecture 16: Scala – Joins

Lecture 17: Python – Joins

Lecture 18: Spark SQL – Joins

Chapter 4: Processing XML, JSON, Delta tables

Lecture 1: Processing Nested XML file

Lecture 2: Processing Nested JSON file

Lecture 3: Delta Table – Time Travel and Vacuum

Chapter 5: Loading data and building ETL data pipeline with dashboard

Lecture 1: Project Description

Lecture 2: Spinning up Azure SQL

Lecture 3: Key Vault

Lecture 4: Secret Scopes

Lecture 5: Project building and mounting of containers

Lecture 6: Reading XML,JSON,CSV and loading to Delta tables & Azure SQL

Lecture 7: Move files from one container to another

Lecture 8: Dashboard

Lecture 9: Azure Data Factory to orchestrate

Lecture 10: Congratulations

Instructors

Priyank Singh
An Engineer who loves to build

Rating Distribution

1 stars: 11 votes
2 stars: 19 votes
3 stars: 68 votes
4 stars: 143 votes
5 stars: 178 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!

Menu