Runs an :py:class:`~mrjob.job.MRJob` on Amazon Elastic MapReduce. Invoked when you run your job with ``-r emr``. :py:class:`EMRJobRunner` run

Wrap a value tagged with !clear in mrjob.conf

Represents steps handled by the script containing your job. Used by :py:meth:`MRJob.steps <mrjob.job.MRJob.steps>`. See :ref:`writing-multi-s

Trivial Job that returns the number of words in each input file

The base class for all MapReduce jobs. See :py:meth:`__init__` for details.

Runs an :py:class:`~mrjob.job.MRJob` in the same process, so it's easy to attach a debugger. This is the default way to run jobs (we assume y

Runs an :py:class:`~mrjob.job.MRJob` on Google Cloud Dataproc. Invoked when you run your job with ``-r dataproc``. :py:class:`DataprocJobRunn

Runs an :py:class:`~mrjob.job.MRJob` on your Hadoop cluster. Invoked when you run your job with ``-r hadoop``. Input and support files can be

Filesystem for Amazon S3 URIs. Typically you will get one of these via ``EMRJobRunner().fs``, composed with :py:class:`~mrjob.fs.ssh.SSHFilesy

Filesystem for Google Cloud Storage (GCS) URIs :param credentials: an optional :py:class:`google.auth.credentials.Credent

UTF-8 encode ``key`` and ``value`` (unicode strings) and join them with a tab character. When reading input, we fall back to latin-1 if we can

Represents the working directory of hadoop/Spark tasks (or bootstrap commands in the cloud). To support Hadoop's distributed cache, paths can

Encode ``(key, value)`` (bytestrings) as ``key`` and ``value`` separated by a tab. If ``key`` or ``value`` is ``None``, don't include a tab.

Recursively return the name and size of each file in the current dir.

Represents a directory on HDFS or S3 where we want to upload local files for consumption by Hadoop. :py:class:`UploadDirManager` tries to giv

Represents a running a custom Jar as a step. Accepts the following keyword arguments: :param jar: The local path to the Jar. On EMR, this ca

Use one of several filesystems depending on the path/URI. This only implements the core :py:class:`~mrjob.fs.base.Filesystem` interface; acce

A simple class to test encoding of objects.

Exception to throw when a step fails. This will automatically be caught and converted to an error message by :py:meth:`mrjob.job.MRJob.run`,

Runs an :py:class:`~mrjob.job.MRJob` locally, for testing purposes. Invoked when you run your job with ``-r local``. Unlike :py:class:`~mrjob

Represents running a Spark step defined in your job. Accepts the following keyword arguments: :param spark: function containing your Spark c

Encode ``(key, value)`` as two reprs separated by a tab. This only works for basic types (we use :py:func:`mrjob.util.safeeval`). .. warning

Runs a :py:class:`~mrjob.job.MRJob` on your Spark cluster (with or without Hadoop). Invoked when you run your job with ``-r spark``. See :ref

Filesystem for URIs accepted by ``hadoop fs``. Typically you will get one of these via ``HadoopJobRunner().fs``, composed with :py:class:`~mrj

Filesystem for local files. Typically you will get one of these via ``MRJobRunner().fs``.

Mock out the *meta* field for various boto3 objects.

Encode ``value`` as a repr and discard ``key`` (``key`` is read in as None). See :py:class:`ReprProtocol` for details.

Represents a running a separate Python script through Spark Accepts the following keyword arguments: :param script: The local path to the Py

Represents a running a separate Jar through Spark Accepts the following keyword arguments: :param jar: The local path to the Python script t

Handle transient errors, with configurable backoff. This class can wrap any object. The wrapped object will behave like the original one, exc

Attempt to UTF-8 decode line (without trailing newline) into ``value``, falling back to latin-1. (``key`` is always ``None``). Output ``value``

Read line (without trailing newline) directly into ``value`` (``key`` is always ``None``). Output ``value`` (bytes) directly, discarding ``key``.

It's a boring job, but somebody had to do it.

Maps word count to a sorted list of words with that count.

Print out ``self.options.runner`` (which should be ``None``) in tasks.

Use Common Crawl .wet files to map from phone number to the most likely URL.

Print out ``self.option.runner`` in tasks.

Mock botocore paginators. Rather than mocking pagination, markers, etc. in every mock API call, we have our API calls return the full results

Encode ``(key, value)`` as two string-escaped pickles separated by a tab. We string-escape the pickles to avoid having to deal with stray

Encode ``value`` as a string-escaped pickle and discard ``key`` (``key`` is read in as ``None``). See :py:class:`PickleProtocol` for details.

Some simple filesystem operations that are common across the local filesystem, S3, GCS, HDFS, and remote machines via SSH. Different runners

A terribly silly way to add a positive integer to values.

Output what directory the mrjob library is in.

Mock out boto3 bucket

Mock out boto3 S3 client :param mock_s3_fs: Maps bucket name to a dictionary with the keys *keys* and *location*. *keys* m

Filesystem for remote systems accessed via SSH. Typically you will get one of these via ``EMRJobRunner().fs``, composed with :py:class:`~mrjob

temporary node created during doctree-read and replaced with a link during doctree-resolved

A simple MRJob that uses the input file option.

A way to check file permissions.