2 Real World Azure Data Engineer Project End to End
2 Real World Azure Data Engineer Project End to End, available at $64.99, has an average rating of 4.27, with 33 lectures, based on 1014 reviews, and has 8865 subscribers.
You will learn about You will learn how to Architect, Design and build a real-world enterprise level data platform solution including multiple services. You will learn design solution using ADF, Azure Function, Databricks, pyspark, Azure Data lake storage Gen 2 (ADLS), Azure SQL Server You will learn how to build a real-world data pipeline in Azure Data Factory (ADF). This course has been taught using 2 real world use case scenarios. You will learn how to transform data using Databricks Notebook Activity in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2 You will learn how to build production ready pipelines and good practices and naming standards You will learn how to integrate Databricks with ADF and send the response back from Databricks to ADF You will learn how to develop the triggered based Azure Function to validate files. You will learn how to create Azure Key vault and use it to store secret credentials and SAS token You will learn how to connect the Azure SQL Database and Databricks cluster using the Key Vault You will learn how to mount he Azure Storage Account in the Databricks to access the files and preform transformation on it. You will learn how to transform the data in the Azure Databricks using the pyspark. This course is ideal for individuals who are Aspiring Data engineer who are searching for project to add in resume or Someone who is looking for Real World uses cases to implement as Data engineering Solution or University students looking for a career in Data Engineering or IT developers working on other disciplines trying to move to Data Engineering or Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies or Data Architects looking to gain an understanding about Azure Data Engineering stack or Data Scientists who want extend their knowledge into data engineering It is particularly useful for Aspiring Data engineer who are searching for project to add in resume or Someone who is looking for Real World uses cases to implement as Data engineering Solution or University students looking for a career in Data Engineering or IT developers working on other disciplines trying to move to Data Engineering or Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies or Data Architects looking to gain an understanding about Azure Data Engineering stack or Data Scientists who want extend their knowledge into data engineering.
Enroll now: 2 Real World Azure Data Engineer Project End to End
Summary
Title: 2 Real World Azure Data Engineer Project End to End
Price: $64.99
Average Rating: 4.27
Number of Lectures: 33
Number of Published Lectures: 33
Number of Curriculum Items: 33
Number of Published Curriculum Objects: 33
Original Price: $19.99
Quality Status: approved
Status: Live
What You Will Learn
- You will learn how to Architect, Design and build a real-world enterprise level data platform solution including multiple services.
- You will learn design solution using ADF, Azure Function, Databricks, pyspark, Azure Data lake storage Gen 2 (ADLS), Azure SQL Server
- You will learn how to build a real-world data pipeline in Azure Data Factory (ADF). This course has been taught using 2 real world use case scenarios.
- You will learn how to transform data using Databricks Notebook Activity in Azure Data Factory (ADF) and load into Azure Data Lake Storage Gen2
- You will learn how to build production ready pipelines and good practices and naming standards
- You will learn how to integrate Databricks with ADF and send the response back from Databricks to ADF
- You will learn how to develop the triggered based Azure Function to validate files.
- You will learn how to create Azure Key vault and use it to store secret credentials and SAS token
- You will learn how to connect the Azure SQL Database and Databricks cluster using the Key Vault
- You will learn how to mount he Azure Storage Account in the Databricks to access the files and preform transformation on it.
- You will learn how to transform the data in the Azure Databricks using the pyspark.
Who Should Attend
- Aspiring Data engineer who are searching for project to add in resume
- Someone who is looking for Real World uses cases to implement as Data engineering Solution
- University students looking for a career in Data Engineering
- IT developers working on other disciplines trying to move to Data Engineering
- Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies
- Data Architects looking to gain an understanding about Azure Data Engineering stack
- Data Scientists who want extend their knowledge into data engineering
Target Audiences
- Aspiring Data engineer who are searching for project to add in resume
- Someone who is looking for Real World uses cases to implement as Data engineering Solution
- University students looking for a career in Data Engineering
- IT developers working on other disciplines trying to move to Data Engineering
- Data Engineers/ Data Warehouse Developers currently working on on-premises technologies, or other cloud platforms such as AWS or GCP who want to learn Azure Technologies
- Data Architects looking to gain an understanding about Azure Data Engineering stack
- Data Scientists who want extend their knowledge into data engineering
This course will help you in preparing and mastering your Azure Data engineering Concepts.
It is not like any random project like covid, or twitter analysis. These project is real world projects on which I personally worked and developed it for big clients.
Highlights of the Course:
-
Designed to keep only précised information no beating around the bush. (To save your time).
-
Real time implementation, no dummy use case.
-
Can be added as part of your resume.
-
It will help you to showcase your experience in interviews and discussion.
-
Involve complex architecture solution which is aligned with industry best practices.
-
Single projects involve various component integration like ADF, ADLS, Databricks, Azure SQL DB, Key Vault.
-
Solves the problem of real time experience for new Data engineers.
This course has been developed in mind to keep all the best practices followed in the Industry as an data engineering project and solution.
Technologies involved:
-
Azure Data Lake Storage Gen 2
-
Azure Data Factory
-
Data Factory Pipeline
-
Azure Functions
-
Azure Key Vault
-
Azure SQL DB
-
SSMS
-
AWS S3 Bucket
-
Connect ADF to Databricks
-
Connect Databricks to SQL Server
-
Connect Databricks to ADLS
-
Connect S3 to Azure Cloud
-
Triggers
-
SAS token
-
Create Secrets scope in Databricks
-
Store secretes in Key Vault and access them
What you will learn after this course:
-
How to think, design and develop the solution in the data engineering world.
-
How to create the architecture diagram for data engineering projects.
-
How to Create Azure Data Factory Account
-
How to Create Azure Data Lake Storage Gen 2 account.
-
How to Create Azure Databricks Workspace.
-
How to create S3 storage account.
-
How to create Azure Function.
-
How to implement logic in the Databricks notebook using pyspark.
-
How to connect ADF to Databricks.
-
How to chain the multiple pieces together in project.
-
How to create Azure SQL Server.
-
How to load the data from file to Azure SQL server.
-
How to connect Databricks notebook with Azure SQL Server.
-
How to Store secrets in the Azure Key Vault.
Course Curriculum
Chapter 1: Project 1: Smart Vehicles
Lecture 1: Understand the Vehicle System Use Case
Lecture 2: Create Architect Diagram for Vehicles Data engineering Solution
Lecture 3: Practical Lab: Create S3 bucket in the AWS Account and upload data.
Lecture 4: Practical Lab: Create ADLS Storage account(Landing Folder)
Lecture 5: Practical Lab: Create Azure Data Factory Account For Data pipelines
Lecture 6: Practical Lab: Create Azure Key Vault and Add the secrets
Lecture 7: Practical Lab: Create Pipeline End to end pipeline with triggers enabled
Lecture 8: Practical Lab: Create Azure Function with blob trigger logic
Lecture 9: Azure Function: Update the JSON Format
Lecture 10: Create Azure SQL Server and Database
Lecture 11: Practical Lab: Add another pipelines for moving data from Staging to SQL DB
Lecture 12: End to End Testing
Chapter 2: Project 2: AP Morgan Data Platform
Lecture 1: Project 2: AP Morgan Data Platform Introduction
Lecture 2: Project 2: Architecture Diagram
Lecture 3: Practical Lab: Create Azure Data Lake Storage account and Drop input files
Lecture 4: Practical Lab: Create Azure Databricks Account Workspace
Lecture 5: Practical Lab: Azure SQL Server
Lecture 6: Practical Lab: Connect SQL DB with SSMS and Create MetaData / Schema table
Lecture 7: Practical Lab: Azure Key Vault to keep SQL server password and SAS token Lab
Lecture 8: Practical Lab: Create a secrets scope in Databricks
Lecture 9: Practical Lab: Create Cluster in Azure Databricks
Lecture 10: Practical Lab: Add notebook in Databricks and Implement the Business Logic
Lecture 11: Practical Lab: Azure Data Factory For AP Morgan
Lecture 12: Practical Lab: Create Azure Databricks Linked Service in ADF
Lecture 13: Practical Lab: Create ADF Pipeline to call Notebook and Test End to End Flow
Chapter 3: Azure Data Factory Quick Walk through
Lecture 1: Azure Data Factory Introduction
Lecture 2: Lab: Create Your First ADF Pipeline
Lecture 3: Lab: Parameterized Dataset in ADF
Chapter 4: Intergration Runtime and Triggers Azure Data Factory
Lecture 1: Integration Runtime
Lecture 2: Lab: Schedule Trigger in ADF
Lecture 3: Lab: Tumbling Window Trigger in ADF
Lecture 4: Lab: Storage Event type Trigger in ADF
Chapter 5: Data Flow in ADF
Lecture 1: Learn Data Flow in Azure Data Factory
Instructors
-
Deepak Goyal
Microsoft Certified Trainer | Architect| Top Voice LinkedIn
Rating Distribution
- 1 stars: 29 votes
- 2 stars: 21 votes
- 3 stars: 111 votes
- 4 stars: 360 votes
- 5 stars: 493 votes
Frequently Asked Questions
How long do I have access to the course materials?
You can view and review the lecture materials indefinitely, like an on-demand channel.
Can I take my courses with me wherever I go?
Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!
You may also like
- Top 10 Language Learning Courses to Learn in November 2024
- Top 10 Video Editing Courses to Learn in November 2024
- Top 10 Music Production Courses to Learn in November 2024
- Top 10 Animation Courses to Learn in November 2024
- Top 10 Digital Illustration Courses to Learn in November 2024
- Top 10 Renewable Energy Courses to Learn in November 2024
- Top 10 Sustainable Living Courses to Learn in November 2024
- Top 10 Ethical AI Courses to Learn in November 2024
- Top 10 Cybersecurity Fundamentals Courses to Learn in November 2024
- Top 10 Smart Home Technology Courses to Learn in November 2024
- Top 10 Holistic Health Courses to Learn in November 2024
- Top 10 Nutrition And Diet Planning Courses to Learn in November 2024
- Top 10 Yoga Instruction Courses to Learn in November 2024
- Top 10 Stress Management Courses to Learn in November 2024
- Top 10 Mindfulness Meditation Courses to Learn in November 2024
- Top 10 Life Coaching Courses to Learn in November 2024
- Top 10 Career Development Courses to Learn in November 2024
- Top 10 Relationship Building Courses to Learn in November 2024
- Top 10 Parenting Skills Courses to Learn in November 2024
- Top 10 Home Improvement Courses to Learn in November 2024