Object that holds the data required to do a query based export of an endpoint. Also exposes some helper functions to assist in generating

Represents an input to mapreduce: a set of files and an input reader that knows how to process them.

All SPL files on S3 (IDs), we only want the most recent versions. This task generates a spl_extract.db that is only the current version.

A task that does absolutely nothing.

Task that loads different CSV files, depending upon what the value of `loader_task`. `init`: process all files except those listed in IGN

A split represents a subset (potentially all) of an input file. Splits allow for processing files that can be read by multiple processes simul

Ignores all outputs and produces no output files.

Creates an index used for SPL version resolution and ID/SET_ID mapping. This task has a cross-dependency upon the SPL pipeline. It returns a li

Convenience class for batching operations. When more than `batch_size` operations have been added to the batch, `fn` will be invoked with `arg

The Rxnorm data needs to be rolled up to the SPL_SET_ID level, so first we clean the data and group it with a ListReducer.

Manages a number of leveldb "shards" (partitions). LevelDB does not support concurrent writers, so we create a separate output shard for each

The UDI document has a unique placement requirement, so we need to override the `harmonize()` method to place the `openfda` section on ea

Mapper for the CSV2JSON map-reduction. There is some special logic in here to generate a hash() from the top level key/value pairs for the id i

There will be some namespace collision since the inputs come from the files listed below and the output of this process will be consumed by

This task downloads all datasets that have not yet been fetched.