Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)

Course Curriculum

Chapter 1: Intro to web crawling and web scraping

Lecture 1: What is the web crawling and web scraping and how is it different ?

Lecture 2: Legality of web scraping and web crawling

Lecture 3: Tools we will be using during development

Lecture 4: Methods of web crawling and web scraping

Lecture 5: Optional: note about deprecation of Request/Request-Promise

Chapter 2: Getting all products from H&M and saving it to mongodb (Method 1)

Lecture 1: Finding hidden API using Chrome Dev Tools

Lecture 2: Testing hidden API inside Postman, and finding other section API endpoints

Lecture 3: Initializing NPM + some info about Nodejs Request and Needle

Lecture 4: Creating our HTTP request with needle inside Nodejs

Lecture 5: Adding User-Agent header to get past denial in nodejs

Lecture 6: Creating MongoDB cluster for saving data

Lecture 7: Connecting to MongoDB cluster from Nodejs

Lecture 8: Saving data to MongoDB

Lecture 9: Getting all products in MongoDB using a loop with offset variable and pagesize

Chapter 3: AliExpress – getting lots of products and prices using Method 1 (hidden API)

Lecture 1: Finding hidden API using Chrome Dev Tools

Lecture 2: Making API request from Postman with correct headers

Lecture 3: Making API request from Nodejs using Fetch API

Lecture 4: Getting many items using a for loop and sleep function

Lecture 5: Saving AliExpress products to MongoDB

Chapter 4: Building a Amazon webcrawler in Nodejs (Method 2, HTTP Requests)

Lecture 1: Intro to project

Lecture 2: Why are we using HTTP requests and not Puppeteer?

Lecture 3: Initializing NPM + installing jest, cheerio and needle npm packages

Lecture 4: Writing our reuseable httpRequest module for our testing and crawling

Lecture 5: Creating our test HTML file (check resources for URL)

Lecture 6: Setting up testing and intro to testing

Lecture 7: Writing our first test for our HTML parser

Lecture 8: Getting title from product page and making our test pass

Lecture 9: Getting the price from product page

Lecture 10: Making our second test and getting product links from page

Lecture 11: Writing out our actual webcrawling in 6 minutes!

Lecture 12: Setup so we only crawl only unique product ID's

Lecture 13: Adding a new test case for different layout + outtro

Chapter 5: Puppeteer web crawling on LinkedIn

Lecture 1: Intro to project

Lecture 2: Initializing project with puppeteer and cheerio packages

Lecture 3: Opening Puppeteer browser and navigating to URL

Lecture 4: Login to Linkedin using Puppeteer

Lecture 5: Getting profile links on a LinkedIn profile

Lecture 6: Building web crawler loop for Puppeteer