Web Scraping with JavaScript: A Beginner’s Guide

Developers are using web scrapers for various types of data fetching. Let us learn how to do web scrape using JavaScript.

NodeJS, the runtime for JavaScript, has made the language one of the most popular and extensively used globally. For both online and mobile applications, JavaScript now offers the necessary tools. This article will demonstrate how NodeJS’s robust ecosystem enables you to scrape the web to effectively satisfy most of your needs.

What is Web Scraping?

Scraping a website’s content and data using bots is web scraping. Web scraping, unlike screen scraping, collects the HTML code and data contained in a database from a website. As a result, the website’s content can be copied to another location. Web scraping is utilized in various digital enterprises that depend on data collection.

Many companies are using web scraping and big data to revolutionize business intelligence. Web scraping is commonly used for the following purposes.

Search engines use bots to crawl a website, assess its content, and rank it in the results. For example, bots automatically retrieve prices and product descriptions for affiliated vendor websites on price comparison sites. Also, market research firms use web scrapers to collect information from forums and social media.

PHP and Python are commonly used for web scraping, but now you can also use web scraping with JavaScript.

Prerequisites

Here are the things you will require before starting web scraping with JavaScript (NodeJS).

• Web browser
• Web page (From which you will extract data)
• Code editors
• js
• Axios
• Cheerio
• Puppeteer

Installation Process

Now that we know about the application, let’s get started with the installation process.

NodeJS

Node.js makes it easy to automate the time-consuming collection of data from websites. Follow these steps to get it installed on your computer: get the software and the installation instructions. In addition to Node.js, npm (the Node Package Manager) will also be downloaded and installed as part of the installation process.

Node.js comes with npm as its default package manager. The consumption of packages will be quick and straightforward, with npm’s support you will be employing packages to facilitate web scraping. Run the installation command (npm init) from inside your project’s root directory to produce a package.json file containing all the project’s information.

Axios

Promise-based HTTP client Axios is available in both Node.js and the browser. If you want to make HTTP requests from Node.js using promises, this npm package will help you. The Axios platform can also handle numerous concurrent queries and automatically transform data into JSON format.

Run the installation command (npm install Axios) from the command line in your project’s directory folder. Your project directory will be immediately established with a node modules folder, where NPM will install Axios.

Cheerio

Cheerio offers a jQuery-like syntax for altering web page content, making it a fast and lightweight module. Consequently, the selection, editing, and viewing of DOM components on a web page are substantially simplified.

Cheerio is an excellent tool for fast parsing and manipulating the DOM. However, it does not behave like a web browser. For example, no JavaScript is executed, no external resources are loaded, and no CSS style is applied.

You can install it by running the installation command (npm install cheerio) on the command line of your project’s directory folder. Like Axios, npm will install Cheerio in a node modules folder, which will be automatically generated in your project’s directory by default.

Puppeteer

As a Node.js library, Puppeteer may be used to manipulate and retrieve data from a headless Chrome browser.

As HTTP-based tools like Axios may not be able to provide the desired results because of JavaScript-based websites, by using Puppeteer, you can run JavaScript like a browser, scrape dynamic material from websites, and replicate the browser experience.

Open your project’s directory in the terminal, then type the installation command (npm install puppeteer) to install it.

Now that the installation process is done let us jump right into web scraping!

Scraping

Let us learn how to use JavaScript to scrape data from a website.

Allow us to utilize a web browser’s inspector feature to locate the specific HTML components that contain the data we are searching for.

The data for the number of comments is included inside an <a> element, which is initiated from the <span> tag and has the class of comment bubbles. This information will be utilized to choose these items on the page using Cheerio.

The procedures for developing the scraping logic are as follows:

1. Begin by creating the index.js file, which will contain the programming logic for getting data from the web page.
2. Then, use the built-in ‘require’ function in Node.js, including the modules that will be used in the project.
3. Now, perform a GET HTTP call using Axios to the target web page. Take note that when a request is made to a web page, it responds. This Axios response object comprises many components, one of which is data referring to the payload delivered by the server. As a result, when a GET request is performed, we output the HTML-formatted data included in the response.

4. Next, populate a Cheerio instance with the response data. In this way, a Cheerio object can be built to assist in parsing the HTML from the destination web page and locating the DOM components containing the data we are looking for. Just like we do with jQuery.
5. The next step will be to utilize Cheerio’s selector syntax to find the items that contain the data we are looking for. Finally, export the data in a text format using the ‘text()’ function.
6. In the end, log the errors that occur throughout the scraping process.

When the appropriate code is performed using the ‘node index.js’ command, it returns the data you requested from the destination web page.

Code:

const axios = require("axios");

const cheerio = require("cheerio");



axios

  .get("your website url")   

//**[for example .get("https://en.wikipedia.org/wiki/Web_scraping")]"**

  .then((response) => {



    const html = response.data;



    const $ = cheerio.load(html);



    const scrapedata = $("a", ".comment-bubble").text();



    console.log(scrapedata);

  })



  .catch((error) => {

    console.log(error);

  });

Conclusion

That is how you can use JavaScript and Node.js for web scraping. You will be able to extract valuable data from websites and incorporate it into your application using these abilities.

If you are looking to develop anything more complex, the documentation for Axios, Cheerio, and Puppeteer may help you get started fast.