Album corresponds to a image album

BasePage corresponds to the most viewed photos page at a given page number

Comment corresponds to the data structure for a PornHub image comment. It contains simple information about the comments. To not reveal too much info

Configuration holds config information necessary to run the PornHub crawler TimeDelaySeconds is the number of seconds to wait between crawling each s

HealthCheckResponse returns status info to check if the crawler is working.

Image corresponds to the data structure of image metadata pulled from PornHub. It references popular comments and tags, as well as general info about

MRDeriveAlbumDataset this MRJob MapReduce job takes the raw_data from crawling pornhub and generates an album specific dataset fromt that

MRDeriveCommentsDataset this MRJob MapReduce class takes in the images dataset and creates a dataset of comments while still holding onto

MRDeriveImageDataset this MRJob MapReduce class takes in the raw_data from crawling pornhub and generates an image specific dataset while

MRDeriveTagsFrequencies this MRJob MapReduce job takes the albums dataset and generates the frequencies of all tags this MR job doesn

MRDeriveTagsGraph this MRJob MapReduce job takes the albums dataset and generates associations between which tags are seen with which othe

Tag corresponds to generic information about tags (just the name currently to save space)

Simple metadata about a user, including their username, number of subscribers, friends, views, etc.