Make a request with retry logic for rate limits

Convert preview.redd.it links to i.redd.it, remove query params, etc.

Generate a descriptive caption for an image using JoyCaption2 and save it to a .txt file. Args: image_path (str): Path to the image

Download all media from a specific thread

Extracts the sharpest frame from a video file based on Laplacian variance, excluding frames that contain significant text or are mostly black

Fetches the HTML content of the given URL.

Return a file extension in lowercase, e.g. '.gif' or '.jpg'

Get the frame rate of the video using ffprobe.

Gather all .json files in scrape_dir and parse them with ingest_file(), skipping filenames listed in completed_list_file. :return: List

Load the set of downloaded thread IDs for each board

Convert a .gif to .mp4 using ffmpeg

Detects whether a frame contains significant text. Uses edge detection and contour analysis to filter frames with text overlays.

Downloads and processes an H264 video stream using FFmpeg.

Download media from threads in a board Args: board: Board name limit: Maximum number of threads to download (None f

Download a file from URL to specified path

Extracts JSON data from the HTML content of the embed page.

Extract all URLs from text using a regex, including markdown-style [text](url) links.

Get the audio codec of the input video.

Selects the best quality stream URL from the master playlist.

Get complete catalog of threads for a board

Return file extension in lowercase

Return a string: 'image', 'gif', 'video', or 'unknown' based on recognized extensions.

Determine file type based on extension Returns: 'image' for .jpg, .jpeg, .png, .webp 'gif' for .gif 'video

Retrieves H264 and AV1 playlists for a given video ID.

Get JSON data for a specific thread

Get the directory path for a specific thread's media

Extract and clean thread title from thread data

Get the video codec of the input video.

Extracts the video ID from the given xHamster video URL.

Initialize and return the Reddit API connection using PRAW. Switch to environment variables (os.getenv) if you want them in .env instead

Detects if a frame is mostly black or white. Args: frame (numpy.ndarray): The video frame. threshold (float): Proportion of p

Check if a thread has been downloaded

Load the 'first_condition_timestamp' from a JSON file if present. Returns a float or None if not available.

Main entry point for the script. Usage: python script.py [NUM_FRAMES or 'auto']

CLI that supports: 1) Scraping subreddits into reddit_data/ 2) Ingesting all .json to gather media URLs 3) Downloading images/gifs/vi

Main function to download xHamster videos based on URL input.

Mark a thread as downloaded

Print download statistics summary

Process all images in a directory and generate captions for each. Args: directory_path (str): Path to the directory containing image

Extract media information from a post

Upload the video file and obtain caption outputs, etc.

Create necessary directories if they don't exist

Setup logging with specified level

Trim the first num_frames from the video at input_path and overwrite the file.

Uploads a local file to the given GCS bucket and returns its GCS URI.

Check if the file is non-empty and optionally verify its metadata. Returns True if valid; otherwise False.

Helper function for multiprocessing. Expects tuple: (input_path, frames_to_trim).