Master Data Engineering using GCP Data Analytics

Master Data Engineering using GCP Data Analytics, available at $89.99, has an average rating of 4.29, with 283 lectures, based on 554 reviews, and has 6115 subscribers.

You will learn about Data Engineering leveraging Services under GCP Data Analytics Setup Development Environment using Visual Studio Code on Windows Building Data Lake using GCS Process Data in the Data Lake using Python and Pandas Build Data Warehouse using Google BigQuery Loading Data into Google BigQuery tables using Python and Pandas Setup Development Environment using Visual Studio Code on Google Dataproc with Remote Connection Big Data Processing or Data Engineering using Google Dataproc Run Spark SQL based applications as Dataproc Jobs using Commands Build Spark SQL based ELT Data Pipelines using Google Dataproc Workflow Templates Run or Instantiate ELT Data Pipelines or Dataproc Workflow Template using gcloud dataproc commands Big Data Processing or Data Engineering using Databricks on GCP Integration of GCS and Databricks on GCP Build and Run Spark based ELT Data Pipelines using Databricks Workflows on GCP Integration of Spark on Dataproc with Google BigQuery Build and Run Spark based ELT Pipeline using Google Dataproc Workflow Template with BigQuery Integration This course is ideal for individuals who are Beginner or Intermediate Data Engineers who want to learn GCP Analytics Services for Data Engineering or Intermediate Application Engineers who want to explore Data Engineering using GCP Analytics Services or Data and Analytics Engineers who want to learn Data Engineering using GCP Analytics Services or Testers who want to learn key skills to test Data Engineering applications built using GCP Analytics Services It is particularly useful for Beginner or Intermediate Data Engineers who want to learn GCP Analytics Services for Data Engineering or Intermediate Application Engineers who want to explore Data Engineering using GCP Analytics Services or Data and Analytics Engineers who want to learn Data Engineering using GCP Analytics Services or Testers who want to learn key skills to test Data Engineering applications built using GCP Analytics Services.

Enroll now: Master Data Engineering using GCP Data Analytics

Summary

Title: Master Data Engineering using GCP Data Analytics

Price: $89.99

Average Rating: 4.29

Number of Lectures: 283

Number of Published Lectures: 283

Number of Curriculum Items: 283

Number of Published Curriculum Objects: 283

Original Price: $199.99

Quality Status: approved

Status: Live

What You Will Learn

Data Engineering leveraging Services under GCP Data Analytics
Setup Development Environment using Visual Studio Code on Windows
Building Data Lake using GCS
Process Data in the Data Lake using Python and Pandas
Build Data Warehouse using Google BigQuery
Loading Data into Google BigQuery tables using Python and Pandas
Setup Development Environment using Visual Studio Code on Google Dataproc with Remote Connection
Big Data Processing or Data Engineering using Google Dataproc
Run Spark SQL based applications as Dataproc Jobs using Commands
Build Spark SQL based ELT Data Pipelines using Google Dataproc Workflow Templates
Run or Instantiate ELT Data Pipelines or Dataproc Workflow Template using gcloud dataproc commands
Big Data Processing or Data Engineering using Databricks on GCP
Integration of GCS and Databricks on GCP
Build and Run Spark based ELT Data Pipelines using Databricks Workflows on GCP
Integration of Spark on Dataproc with Google BigQuery
Build and Run Spark based ELT Pipeline using Google Dataproc Workflow Template with BigQuery Integration

Who Should Attend

Beginner or Intermediate Data Engineers who want to learn GCP Analytics Services for Data Engineering
Intermediate Application Engineers who want to explore Data Engineering using GCP Analytics Services
Data and Analytics Engineers who want to learn Data Engineering using GCP Analytics Services
Testers who want to learn key skills to test Data Engineering applications built using GCP Analytics Services

Target Audiences

Beginner or Intermediate Data Engineers who want to learn GCP Analytics Services for Data Engineering
Intermediate Application Engineers who want to explore Data Engineering using GCP Analytics Services
Data and Analytics Engineers who want to learn Data Engineering using GCP Analytics Services
Testers who want to learn key skills to test Data Engineering applications built using GCP Analytics Services

Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using GCP Data Analytics Stack. It includes services such as Google Cloud Storage, Google BigQuery, GCP Dataproc, Databricks on GCP, and many more.

As part of this course, first you will go ahead and setup environment to learn using VS Code on Windows and Mac.
Once the environment is ready, you need to sign up for Google Cloud Account. We will provide all the instructions to sign up for Google Cloud Account including reviewing billing as well as getting USD 300 Credit.
We typically use Cloud Object Storage as Data Lake. As part of this course, you will learn how to use Google Cloud Storage as Data Lake along with how to manage the files in Google Cloud Storage both by using commands as well as Python. It also covers, integration of Pandas with files in Google Cloud Storage.
GCP provides RDBMS as service via Cloud SQL. You will learn how to setup Postgresql Database Server using Cloud SQL. Once the Database Server is setup, you will also take care of setting up required application database and user. You will also understand how to develop Python based applications by integrating with GCP Secretmanager to retrieve the credentials.
One of the key usage of Data is nothing but building reports and dashboards. Typically reports and dashboards are built using reporting tools pointing to Data Warehouse. As part of Google Data Analytics Services, BigQuery can be used as Data Warehouse. You will learn the features of BigQuery as a Data Warehouse along with key integrations using Python and Pandas.
At times, we need to process heavy volumes of data which also known as Big Data Processing. GCP Dataproc is a fully manage Big Data Service with Hadoop, Spark, Kafka, etc. You will not only learn how to setup the GCP Dataproc cluster, but also you will learn how to use single node Dataproc cluster for the development. You will setup development environment using VS Code with remote connection to the Dataproc Cluster.
Once you understand how to get started with Big Data Processing using Dataproc, you will take care of building end to end ELT Data Pipelines using Dataproc Workflow Templates. You will learn all key commands to submit Dataproc Jobs as well as Workflows. You will end up building ELT Pipelines using Spark SQL.
While Dataproc is GCP Native Big Data Service, Databricks is another prominent Big Data Service available in GCP. You will also understand how to get started with Databricks on GCP.
Once you go through the details about how to get started with Databricks on GCP, you will take care of building end to end ELT Datapipelins using Databricks Jobs and Workflows.
Towards the end of the course you should be fairly comfortable with BigQuery for Data Warehouse and GCP Dataproc for Data Processing, you will learn how to integrate these two key services by building end to end ELT Data Pipeline using Dataproc Workflow. You will also understand how to include Pyspark based application with Spark BigQuery connector as part of the Pipeline.
In the process of building Data Pipelines, you will also revise application development life cycle of Spark, troubleshooting issues related to the spark using relevant web interfaces such as YARN Timeline Server, Spark UI, etc.

Course Curriculum

Chapter 1: Introduction to Data Engineering using GCP Data Analytics

Lecture 1: Introduction to Data Engineering using GCP Data Analytics

Lecture 2: Pre-requisites for Data Engineering using GCP Data Analytics

Lecture 3: Highlights of the Data Engineering using GCP Data Analytics Course

Lecture 4: Overview of Udemy Platform to take course effectively

Lecture 5: Refund Policy and Request for Rating and Feedback

Chapter 2: Setup Environment for Data Engineering using GCP Data Analytics

Lecture 1: Introduction to Setup Environment for Data Engineering using GCP Data Analytics

Lecture 2: Setup VS Code on Windows

Lecture 3: Setup Python 3.9 on Windows

Lecture 4: Configure Environment Variable PATH for Python on Windows

Lecture 5: Integrate VSCode with Python on Windows

Lecture 6: Download Git Repo on to our Local Machines

Lecture 7: Review Data Engineering on GCP Folder

Lecture 8: Setup VS Code Workspace for Data Engineering on GCP

Lecture 9: Setup and Integrate Python 3.9 venv with VS Code Workspace

Lecture 10: Setup and Integrate Python 3.9 venv with VS Code Workspace on Windows

Lecture 11: Conclusion to Setup Environment for Data Engineering using GCP Data Analytics

Chapter 3: Getting Started with GCP for Data Engineering using GCP Data Analytics

Lecture 1: Introduction to Getting Started with GCP

Lecture 2: Pre-requisite Skills to Sign up for course on GCP Data Analytics

Lecture 3: Overview of Cloud Platforms

Lecture 4: Overview of Google Cloud Platform or GCP

Lecture 5: Overview of Signing for GCP Account

Lecture 6: Create New Google Account using Non Gmail Id

Lecture 7: Sign up for GCP using Google Account

Lecture 8: Overview of GCP Credits

Lecture 9: Overview of GCP Project and Billing

Lecture 10: Overview of Google Cloud Shell

Lecture 11: Install Google Cloud SDK on Windows

Lecture 12: Initialize gcloud CLI using GCP Project

Lecture 13: Reinitialize Google Cloud Shell with Project id

Lecture 14: Overview of Analytics Services on GCP

Lecture 15: Conclusion to Get Started with GCP for Data Engineering

Chapter 4: Setting up Data Lake using Google Cloud Storage

Lecture 1: Getting Started with Google Cloud Storage or GCS

Lecture 2: Overview of Google Cloud Storage or GCS Web UI

Lecture 3: Create GCS Bucket using GCP Web UI

Lecture 4: Upload Folders and Files using into GCS Bucket using GCP Web UI

Lecture 5: Review GCS Buckets and Objects using gsutil commands

Lecture 6: Delete GCS Bucket using Web UI

Lecture 7: Setup Data Repository in Google Cloud Shell

Lecture 8: Overview of Data Sets

Lecture 9: Managing Buckets in GCS using gsutil

Lecture 10: Copy Data Sets into GCS using gsutil

Lecture 11: Cleanup Buckets in GCS using gsutil

Lecture 12: Exercise to Manage Buckets and Files in GCS using gsutil

Lecture 13: Overview of Setting up Data Lake using GCS

Lecture 14: Setup Google Cloud Libraries in Python Virtual Environment

Lecture 15: Setup Bucket and Files in GCS using gsutil

Lecture 16: Getting Started to manage files in GCS using Python

Lecture 17: Setup Credentials for Python and GCS Integration

Lecture 18: Review Methods in Google Cloud Storage Python library

Lecture 19: Get GCS Bucket Details using Python

Lecture 20: Manage Blobs or Files in GCS using Python

Lecture 21: Project Problem Statement to Manage Files in GCS using Python

Lecture 22: Design to Upload multiple files into GCS using Python

Lecture 23: Get File Names to upload into GCS using Python glob and os

Lecture 24: Upload all Files to GCS as blobs using Python

Lecture 25: Validate Files or Blobs in GCS using Python

Lecture 26: Overview of Processing Data in GCS using Pandas

Lecture 27: Convert Data to Parquet and Write to GCS using Pandas

Lecture 28: Design to Upload multiple files into GCS using Pandas

Lecture 29: Get File Names to upload into GCS using Python glob and os

Lecture 30: Overview of Parquet File Format and Schemas JSON File

Lecture 31: Get Column Names for Dataset using Schemas JSON File

Lecture 32: Upload all Files to GCS as Parquet using Pandas

Lecture 33: Perform Validation of Files Copied using Pandas

Chapter 5: Setup Postgres Database using Cloud SQL

Lecture 1: Overview of GCP Cloud SQL

Lecture 2: Setup Postgres Database Server using GCP Cloud SQL

Lecture 3: Configure Network for Cloud SQL Postgres Database

Lecture 4: Install Postgres 14 on Windows 11

Lecture 5: Getting Started with pgAdmin on Windows

Lecture 6: Getting Started with pgAdmin on Mac

Lecture 7: Validate Client Tools for Postgres on Mac or PC

Lecture 8: Setup Database in GCP Cloud SQL Postgres Database Server

Lecture 9: Setup Tables in GCP Cloud SQL Postgres Database

Lecture 10: Validate Data in GCP Cloud SQL Postgres Database Tables

Lecture 11: Integration of GCP Cloud SQL Postgres with Python

Lecture 12: Overview of Integration of GCP Cloud SQL Postgres with Pandas

Lecture 13: Read Data From Files to Pandas Data Frame

Lecture 14: Process Data using Pandas Dataframe APIs

Lecture 15: Write Pandas Dataframe into Postgres Database Table

Lecture 16: Validate Data in Postgres Database Tables using Pandas

Lecture 17: Getting Started with Secrets using GCP Secret Manager

Lecture 18: Configure Access to GCP Secret Manager via IAM Roles

Lecture 19: Install Google Cloud Secret Manager Python Library

Lecture 20: Get Secret Details from GCP Secret Manager using Python

Lecture 21: Connect to Database using Credentials from Secret Manager

Lecture 22: Stop GCP Cloud SQL Postgres Database Server

Chapter 6: Build Data Warehouse using Google Big Query

Lecture 1: Overview of Google BigQuery

Lecture 2: Overview of Google BigQuery

Lecture 3: Getting Started with Google BigQuery

Lecture 4: Overview of CRUD Operations in Google BigQuery

Lecture 5: Merge or Upsert into Google BigQuery Tables

Lecture 6: Create Dataset and Tables in Google BigQuery using UI

Lecture 7: Create Table in Google BigQuery using Command

Lecture 8: Exercise to create tables in Google BigQuery

Instructors

Durga Viswanatha Raju Gadiraju
CEO at ITVersity and CTO at Analytiqs, Inc
Pratik Kumar
Sathvika Dandu
Madhuri Gadiraju
Sai Varma
Phani Bhushan Bozzam

Rating Distribution

1 stars: 7 votes
2 stars: 5 votes
3 stars: 38 votes
4 stars: 200 votes
5 stars: 304 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!

Menu