Introduction:

Web scraping is the process of extracting data from websites. However, some websites use JavaScript to load content dynamically, making it difficult to scrape data using traditional methods like using requests or the urllib library

In this tutorial, we will learn how to scrape websites using Selenium, a popular web testing framework.

Flow Diagram

Prerequisites:

Python 3.8

Ubuntu machine

Steps:

Step 1: Create a Virtual environment.

Install the Python venv package using apt.

sudo apt update
sudo apt install python3-venv

After installing the venv package, create the virtual environment.

python3 -m venv .venv

Then activate the virtual environment

Source .venv/bin/activate

source .venv/bin/activate

Step 2: Install Selenium.

Install the Selenium package using the pip package manager by running the following command.

pip install selenium

Step 4: Setting up Selenium

Selenium needs a driver to interact with a browser. In this tutorial, we will use Selenium with the Chrome browser, so we’ll need to install the Chrome driver.

Run the following command to install the Chrome web driver

sudo apt install chromium-chromedriver

Step 5: Scraping Dynamic Websites.

Create the driver instance by calling the webdriver

from selenium import webdriver

# Set up the Chrome driver
driver = webdriver.Chrome()

Now you should see an empty browser open on your screen.

Then navigate to the website that needs to be scraped. In this blog, I am trying to scrape Amazon products using an Amazon URL.

product_url = "https://amzn.eu/d/89BULvG"
driver.get(product_url)

Now you can see the browser is loading the website with the URL, "https://amzn.eu/d/89BULvG".

Now wait for a few seconds, so that the site will load the dynamic content.

From selenium.webdriver.support.ui import WebDriverWait
wait = WebDriverWait(driver, 10)

After the wait, we navigate to the reviews page of the product

driver.find_element(By.XPATH, "//*[text()='See all reviews']").click()

Then we scrape the review data from the div using XPATH and TAG.

Finally, after the data is scraped and processed, the driver can be closed.

driver.quit()

Now the browser window will be closed.

Example Scanning Code:

From selenium import webdriver
from selenium.webdriver.common.by import 
from selenium.webdriver.support.ui import WebDriverWait
from selenium. common.exceptions import NoSuchElementException

from bs4 import BeautifulSoup

# Set up the Chrome driver
driver = webdriver.Chrome()

# access the web page
product_url = "https://amzn.eu/d/89BULvG"
driver.get(product_url)

# driver waits for a certain time for the page to load
wait = WebDriverWait(driver, 10)

soup = BeautifulSoup(driver.page_source, 'html.parser')

review_body = soup.find("div", class_="review-views")

rev_list = review_body.find_all('div', class_='review')
review_list = []
while True:
    Try:
        For review in rev_list:
            reviewer_name = review.find('span', class_='a-profile-name')
            review_rating = float((review.find('span', class_='a-icon-alt').text).split()[0])
            review_summary = review.find('a', class_='review-title').text.strip()
            review_text = review.find('span', class_='review-text').span.text.strip()
            review_date = review.find('span', class_='review-date').text.split('on')[-1].strip()
            review_country = review.find('span', class_='review-date').text.split('on')[0].split()[2]
            data_dict = {
                'reviewer_name': reviewer_name,
                'review_rating': review_rating,
                'review_summary': review_summary,
                'review_text': review_text,
                'review_date': review_date,
                'review_country': review_country
            }
            review_list.append(data_dict)
        next_page = driver.find_element(By.XPATH, "//*[text()='Next page']")
        next_page.click()
        import time
        time.sleep(5)
    # except NoSuchElementException:
    Except Exception:
        break

print(review_list)

In this blog, we have set up the Chrome driver using Selenium and navigated to the product’s web page. Then we used Selenium's WebDriverWait function to wait for a certain time for the page to fully load. Once the page is loaded, we scraped its text using the BeautifulSoup library.

Result:

In this tutorial, we learned how to scrape dynamic websites using Selenium. By using Selenium's built-in functionality to wait for the page to load completely, we can easily scrape data from websites that use JavaScript to load content dynamically.

Scraping dynamic websites using Selenium