EN VI

Python - Scraping Sony camera webpage cant access Specification tab?

How to Python - Scraping Sony camera webpage cant access Specification tab

URL = https://electronics.sony.com/imaging/interchangeable-lens-cameras/full-frame/p/ilce7rm4-b

I need to scrape specifications tab from each camera. It loads after first tapping specification and then see more How to load this data with selenium python script.

I tried

def specs_see_more(driver):
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH,
                                        '//*[@id="PDPSpecificationsLink"]')))
    time.sleep(3)
    see_more_button = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH,
                                    '//*[@id="PDPSpecificationsLink"]')))

    driver.execute_script("arguments[0].scrollIntoView(true);", see_more_button)

    WebDriverWait(driver, 10).until(
        EC.visibility_of(see_more_button))

    see_more_button.click()

    spec_page_source = driver.page_source
    driver_soup = BeautifulSoup(spec_page_source, "html.parser")
    print(driver_soup)

This first clicking specification button and then opening see more but it did not work.

Solution:

see_more_button = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH,
                                    '//*[@id="PDPSpecificationsLink"]')))

In the above line of code, XPATH expression to locate See More button is incorrect.

Refer the following working code to click on See More button within "Specification" tab:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://electronics.sony.com/imaging/interchangeable-lens-cameras/full-frame/p/ilce7rm4-b")
wait = WebDriverWait(driver, 10)

# Below line will click on 'Specifications' tab
wait.until(EC.element_to_be_clickable((By.ID, "PDPSpecificationsLink"))).click()
# Below line will click on 'See More' button
wait.until(EC.element_to_be_clickable((By.XPATH, "(//button[contains(text(),'See More')])[2]"))).click()
time.sleep(5)