- Data Included: fan movie reviews, rating score, review date
In the following code, we employed Selenium to scrape movie reviews, ratings, and review dates for Disney Princess movies from Rotten Tomatoes. First, we set up the Selenium WebDriver using ChromeDriverManager and configured it with a page load timeout to ensure smooth handling of webpage interactions. We navigated to the specified URL and waited for the review section to load using WebDriverWait, ensuring the necessary elements were present before proceeding. To handle paginated reviews, we implemented a loop to click the “Load More” button multiple times, allowing us to access as many reviews as possible within a defined limit.
After loading the reviews, we extracted data for each review element, including the review text, review date, and rating score. The review text was cleaned using regex to remove unwanted characters, while the score was retrieved directly from the rating attribute. Each review was stored in a dictionary and added to a list, ensuring duplicates were avoided. Finally, the collected data was saved into a Pandas DataFrame and exported as a CSV file for further analysis. This automated process efficiently handled dynamic content and pagination, enabling the retrieval of comprehensive review data for analysis.
Show Code
from selenium import webdriverfrom selenium.webdriver.chrome.service import Servicefrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.chrome.options import Optionsimport pandas as pdimport timeimport refrom webdriver_manager.chrome import ChromeDriverManagerdef scrape_reviews(url, output_csv_path="./data/scraped_reviews.csv", max_load_more_clicks=100, review_limit=2000):# # Step 1: Set up Chrome Service with extended read timeout# # Install and set up ChromeDriver using webdriver_manager# chrome_service = Service(ChromeDriverManager().install())# chrome_service.startup_timeout = 600 # Set startup timeout to 600 seconds# # Configure Chrome options and initialize the browser# options = Options()# browser = webdriver.Chrome(service=chrome_service, options=options)# # Step 2: Set page load timeout# # Set a page load timeout of 600 seconds# browser.set_page_load_timeout(600)# # Open the specified URL# browser.get(url) browser = webdriver.Chrome() browser.set_page_load_timeout(600) browser.get(url)# Step 3: Wait for the page to loadtry:# Wait for the audience reviews to be present on the page (max 10 seconds) WebDriverWait(browser, 10).until( EC.presence_of_element_located((By.CLASS_NAME, "audience-reviews__review")) )exceptExceptionas e:# Handle any errors during page loadprint("Error loading reviews:", e) browser.quit()return# Step 4: Initialize variables and lists all_reviews = [] # List to store all scraped reviews load_more_clicks =0# Counter for number of 'Load More' button clicks# Step 5: Click "Load More" buttons up to the maximum number of clickswhile load_more_clicks < max_load_more_clicks:try:# Wait for the 'Load More' button to be clickable (max 30 seconds) load_more_button = WebDriverWait(browser, 10).until( EC.element_to_be_clickable((By.CSS_SELECTOR, "rt-button[data-qa='load-more-btn']")) ) load_more_button.click() # Click the 'Load More' button load_more_clicks +=1print(f"Clicked 'Load More' button {load_more_clicks} times.") time.sleep(5) # Additional wait time to ensure complete loadingexceptExceptionas e:# Handle any errors or when no more 'Load More' button is foundprint("No more 'Load More' button found or an error occurred:", e)break# Step 6: Scrape reviewstry:# Locate all review elements on the current page reviews = browser.find_elements(By.CLASS_NAME, "review-text-container")# Extract review text, date, and score from each review blockfor review in reviews:iflen(all_reviews) >= review_limit:break# Stop if we've reached the review limittry:# Extract the review text review_text = review.find_element(By.CSS_SELECTOR, "p.audience-reviews__review").text.strip()# Preprocess the review text to keep only specified characters review_text = re.sub(r"[^a-zA-Z0-9!?\"'() ]", "", review_text)# Extract the review date review_date = review.find_element(By.CSS_SELECTOR, "span.audience-reviews__duration").text.strip()# Extract the review score score_element = review.find_element(By.TAG_NAME, "rating-stars-group") score = score_element.get_attribute("score")# Create a dictionary for the review data review_data = {"review_text": review_text,"date": review_date,"score": score }# Avoid duplicates and add the review to the listif review_data notin all_reviews: all_reviews.append(review_data)exceptExceptionas e:# Handle any errors during review processingprint(f"Error processing review: {e}")exceptExceptionas e:# Handle any errors during scrapingprint("Error during scraping:", e)# Step 7: Close the browser browser.quit()# Step 8: Save reviews to a CSV file# Create a DataFrame from the scraped reviews and save to CSV df = pd.DataFrame(all_reviews) df.to_csv(output_csv_path, index=False)print(f"Scraped {len(all_reviews)} reviews and saved to '{output_csv_path}'")
- includes: movie release year, director, casts, IMDB rating, movie plot, box offices etc.
In the following code, we utilized the Cinemagoer library to programmatically retrieve detailed information about Disney Princess movies from IMDb. We began by creating an IMDb instance to facilitate movie searches and data retrieval. A predefined list of Disney Princess movies with their respective release years was used to ensure accurate matching. For each movie in the list, we searched IMDb’s database and identified the correct movie by strictly matching both the title and release year. Once a match was found, we used the movie’s unique ID to fetch detailed information, which included metadata such as genres, directors, cast members, ratings, and votes.
The retrieved movie details were then structured into a dictionary for each title, extracting relevant fields such as top cast members, runtime, plot summary, box office performance, and production details like countries and languages. This structured data was converted into a Pandas DataFrame for further analysis and visualization. Finally, the DataFrame was exported as a CSV file, allowing for easy integration into subsequent stages of our analysis pipeline. This process efficiently automated the extraction of rich metadata for Disney Princess movies, providing a solid foundation for further exploration and insights.
Show Code
from imdb import Cinemagoer# Create an IMDb instance to search and retrieve movie data:ia = Cinemagoer()
Show Code
# List of movies and their release yearsmovies = [ {"title": "Snow White and the Seven Dwarfs", "year": 1937}, {"title": "Cinderella", "year": 1950}, {"title": "Sleeping Beauty", "year": 1959}, {"title": "The Little Mermaid", "year": 1989}, {"title": "Beauty and the Beast", "year": 1991}, {"title": "Aladdin", "year": 1992}, {"title": "Pocahontas", "year": 1995}, {"title": "Mulan", "year": 1998}, {"title": "The Princess and the Frog", "year": 2009}, {"title": "Tangled", "year": 2010}, {"title": "Brave", "year": 2012}, {"title": "Frozen", "year": 2013}, {"title": "Frozen II", "year": 2019}, # Adding the second Frozen movie {"title": "Moana", "year": 2016},]# Dictionary to store movie datamovie_data = {}# Loop through the movies to fetch datafor movie in movies: search_results = ia.search_movie(movie["title"])for result in search_results:# Match title and year strictlyif ( result["title"].lower() == movie["title"].lower()and"year"in resultandint(result["year"]) == movie["year"] ):# Get detailed movie information movie_id = result.movieID movie_details = ia.get_movie(movie_id) movie_data[movie["title"]] = movie_detailsprint(f"Retrieved details for {movie['title']} ({movie['year']})")break
Retrieved details for Snow White and the Seven Dwarfs (1937)
Retrieved details for Cinderella (1950)
Retrieved details for Sleeping Beauty (1959)
Retrieved details for The Little Mermaid (1989)
Retrieved details for Beauty and the Beast (1991)
Retrieved details for Aladdin (1992)
Retrieved details for Pocahontas (1995)
Retrieved details for Mulan (1998)
Retrieved details for The Princess and the Frog (2009)
Retrieved details for Tangled (2010)
Retrieved details for Brave (2012)
Retrieved details for Frozen (2013)
Retrieved details for Frozen II (2019)
Retrieved details for Moana (2016)
Show Code
import pandas as pd# Extract relevant fields and create a DataFramemovie_details_list = []for title, details in movie_data.items(): movie_details_list.append({"title": title,"year": details.get('year'),"genres": details.get('genres'),"directors": [director['name'] for director in details.get('directors', [])],"cast": [person['name'] for person in details.get('cast', [])[:5]], # Top 5 cast members"rating": details.get('rating'),"votes": details.get('votes'),"plot": details.get('plot outline'),"box_office": details.get("box office", {}),"countries": details.get("countries", []),"languages": details.get("languages", []),"runtimes": details.get("runtimes", []) })df_movies = pd.DataFrame(movie_details_list)
Show Code
# Save to CSVdf_movies.to_csv("../../data/raw-data/imdb_movie_details.csv", index=False)