Bulk Downloader for Reddit

This is a tool to download submissions or submission data from Reddit. It can be used to archive data or even crawl Reddit to gather research data. The BDFR is flexible and can be used in scripts if needed through an extensive command-line interface. List of currently supported sources

If you wish to open an issue, please read the guide on opening issues to ensure that your issue is clear and contains everything it needs to for the developers to investigate.

Included in this README are a few example Bash tricks to get certain behaviour. For that, see Common Command Tricks.

Installation

Bulk Downloader for Reddit needs Python version 3.9 or above. Please update Python before installation to meet the requirement.

Then, you can install it via pip with:

python3 -m pip install bdfr --upgrade

or via pipx with:

python3 -m pipx install bdfr

To update BDFR, run the above command again for pip or pipx upgrade bdfr for pipx installations.

To check your version of BDFR, run bdfr --version

To install shell completions, run bdfr completions

AUR Package

If on Arch Linux or derivative operating systems such as Manjaro, the BDFR can be installed through the AUR.

• Latest Release: https://aur.archlinux.org/packages/python-bdfr
• Latest Development Build: https://aur.archlinux.org/packages/python-bdfr-git

Source code

If you want to use the source code or make contributions, refer to CONTRIBUTING

Usage

The BDFR works by taking submissions from a variety of "sources" from Reddit and then parsing them to download. These sources might be a subreddit, multireddit, a user list, or individual links. These sources are combined and downloaded to disk, according to a naming and organisational scheme defined by the user.

There are three modes to the BDFR: download, archive, and clone. Each one has a command that performs similar but distinct functions. The download command will download the resource linked in the Reddit submission, such as the images, video, etc. The archive command will download the submission data itself and store it, such as the submission details, upvotes, text, statistics, as and all the comments on that submission. These can then be saved in a data markup language form, such as JSON, XML, or YAML. Lastly, the clone command will perform both functions of the previous commands at once and is more efficient than running those commands sequentially.

Note that the clone command is not a true, failthful clone of Reddit. It simply retrieves much of the raw data that Reddit provides. To get a true clone of Reddit, another tool such as HTTrack should be used.

After installation, run the program from any directory as shown below:

bdfr download

bdfr archive

bdfr clone

However, these commands are not enough. You should chain parameters in Options according to your use case. Don't forget that some parameters can be provided multiple times. Some quick reference commands are:

bdfr download ./path/to/output --subreddit Python -L 10

bdfr download ./path/to/output --user reddituser --submitted -L 100

bdfr download ./path/to/output --user me --saved --authenticate -L 25 --file-scheme '{POSTID}'

bdfr download ./path/to/output --subreddit 'Python, all, mindustry' -L 10 --make-hard-links

bdfr archive ./path/to/output --user reddituser --submitted --all-comments --comment-context

bdfr archive ./path/to/output --subreddit all --format yaml -L 500 --folder-scheme ''

Alternatively, you can pass options through a YAML file.

bdfr download ./path/to/output --opts my_opts.yaml

For example, running it with the following file

skip: [mp4, avi]
file_scheme: "{UPVOTES}_{REDDITOR}_{POSTID}_{DATE}"
limit: 10
sort: top
subreddit:
  - EarthPorn
  - CityPorn

would be equilavent to (take note that in YAML there is file_scheme instead of file-scheme):

bdfr download ./path/to/output --skip mp4 --skip avi --file-scheme "{UPVOTES}_{REDDITOR}_{POSTID}_{DATE}" -L 10 -S top --subreddit EarthPorn --subreddit CityPorn

Any option that can be specified multiple times should be formatted like subreddit is above.

In case when the same option is specified both in the YAML file and in as a command line argument, the command line argument takes priority

Options

The following options are common between both the archive and download commands of the BDFR.

• directory
  • This is the directory to which the BDFR will download and place all files
• --authenticate
  • This flag will make the BDFR attempt to use an authenticated Reddit session
  • See Authentication for more details
• --config
  • If the path to a configuration file is supplied with this option, the BDFR will use the specified config
  • See Configuration Files for more details
• --opts
  • Load options from a YAML file.
  • Has higher prority than the global config file but lower than command-line arguments.
  • See opts_example.yaml for an example file.
• --disable-module
  • Can be specified multiple times
  • Disables certain modules from being used
  • See Disabling Modules for more information and a list of module names
• --filename-restriction-scheme
  • Can be: windows, linux
  • Turns off the OS detection and specifies which system to use when making filenames
  • See Filesystem Restrictions
• --ignore-user
  • This will add a user to ignore
  • Can be specified multiple times
• --include-id-file
  • This will add any submission with the IDs in the files provided
  • Can be specified multiple times
  • Format is one ID per line
• --log
  • This allows one to specify the location of the logfile
  • This must be done when running multiple instances of the BDFR, see Multiple Instances below
• --saved
  • This option will make the BDFR use the supplied user's saved posts list as a download source
  • This requires an authenticated Reddit instance, using the --authenticate flag, as well as --user set to me
• --search
  • This will apply the input search term to specific lists when scraping submissions
  • A search term can only be applied when using the --subreddit and --multireddit flags
• --submitted
  • This will use a user's submissions as a source
  • A user must be specified with --user
• --upvoted
  • This will use a user's upvoted posts as a source of posts to scrape
  • This requires an authenticated Reddit instance, using the --authenticate flag, as well as --user set to me
• -L, --limit
  • This is the limit on the number of submissions retrieve
  • Default is max possible
  • Note that this limit applies to each source individually e.g. if a --limit of 10 and three subreddits are provided, then 30 total submissions will be scraped
  • If it is not supplied, then the BDFR will default to the maximum allowed by Reddit, roughly 1000 posts. We cannot bypass this.
• -S, --sort
  • This is the sort type for each applicable submission source supplied to the BDFR
  • This option does not apply to upvoted or saved posts when scraping from these sources
  • The following options are available:
    • controversial
    • hot (default)
    • new
    • relevance (only available when using --search)
    • rising
    • top
• -l, --link
  • This is a direct link to a submission to download, either as a URL or an ID
  • Can be specified multiple times
• -m, --multireddit
  • This is the name of a multireddit to add as a source
  • Can be specified multiple times
    • This can be done by using -m multiple times
    • Multireddits can also be used to provide CSV multireddits e.g. -m 'chess, favourites'
  • The specified multireddits must all belong to the user specified with the --user option
• -s, --subreddit
  • This adds a subreddit as a source
  • Can be used mutliple times
    • This can be done by using -s multiple times
    • Subreddits can also be used to provide CSV subreddits e.g. -m 'all, python, mindustry'
• -t, --time
  • This is the time filter that will be applied to all applicable sources
  • This option does not apply to upvoted or saved posts when scraping from these sources
  • This option only applies if sorting by top or controversial. See --sort for more detail.
  • The following options are available:
    • all (default)
    • hour
    • day
    • week
    • month
    • year
  • --time-format
    • This specifies the format of the datetime string that replaces {DATE} in file and folder naming schemes
    • See Time Formatting Customisation for more details, and the formatting scheme
• -u, --user
  • This specifies the user to scrape in concert with other options
  • When using --authenticate, --user me can be used to refer to the authenticated user
  • Can be specified multiple times for multiple users
    • If downloading a multireddit, only one user can be specified
• -v, --verbose
  • Increases the verbosity of the program
  • Can be specified multiple times

Downloader Options

The following options apply only to the download command. This command downloads the files and resources linked to in the submission, or a text submission itself, to the disk in the specified directory.

• --make-hard-links
  • This flag will create hard links to an existing file when a duplicate is downloaded in the current run
  • This will make the file appear in multiple directories while only taking the space of a single instance
• --max-wait-time
  • This option specifies the maximum wait time for downloading a resource
  • The default is 120 seconds
  • See Rate Limiting for details
• --no-dupes
  • This flag will not redownload files if they were already downloaded in the current run
  • This is calculated by MD5 hash
• --search-existing
  • This will make the BDFR compile the hashes for every file in directory
  • The hashes are used to remove duplicates if --no-dupes is supplied or make hard links if --make-hard-links is supplied
• --file-scheme
  • Sets the scheme for files
  • Default is {REDDITOR}_{TITLE}_{POSTID}
  • See Folder and File Name Schemes for more details
• --folder-scheme
  • Sets the scheme for folders
  • Default is {SUBREDDIT}
  • See Folder and File Name Schemes for more details
• --exclude-id
  • This will skip the download of any submission with the ID provided
  • Can be specified multiple times
• --exclude-id-file
  • This will skip the download of any submission with any of the IDs in the files provided
  • Can be specified multiple times
  • Format is one ID per line
• --skip-domain
  • This adds domains to the download filter i.e. submissions coming from these domains will not be downloaded
  • Can be specified multiple times
  • Domains must be supplied in the form example.com or img.example.com
• --skip
  • This adds file types to the download filter i.e. submissions with one of the supplied file extensions will not be downloaded
  • Can be specified multiple times
• --skip-subreddit
  • This skips all submissions from the specified subreddit
  • Can be specified multiple times
  • Also accepts CSV subreddit names
• --min-score
  • This skips all submissions which have fewer than specified upvotes
• --max-score
  • This skips all submissions which have more than specified upvotes
• --min-score-ratio
  • This skips all submissions which have lower than specified upvote ratio
• --max-score-ratio
  • This skips all submissions which have higher than specified upvote ratio

Archiver Options

The following options are for the archive command specifically.

• --all-comments
  • When combined with the --user option, this will download all the user's comments
• -f, --format
  • This specifies the format of the data file saved to disk
  • The following formats are available:
    • json (default)
    • xml
    • yaml
• --comment-context
  • This option will, instead of downloading an individual comment, download the submission that comment is a part of
  • May result in a longer run time as it retrieves much more data

Cloner Options

The clone command can take all the options listed above for bot