Redirect stdin, stdout, and stderr for automated testing. You can set stdin, stdout, and stderr to file objects. By default, they'll

Join *paths* onto *path* (which may be a URI)

Entry point for running job from the command-line. This is also the entry point when a mapper or reducer is run by Hadoop Streaming.

Add the given patcher to this test case's cleanup actions, then start it, and return the mock it returns. Example: mock_turtle = self

Stream the job's output, as a stream of ``bytes``. If there are multiple output files, there will be an empty bytestring (``b''``) bet

Recursively list all files in the given path. We don't return directories for compatibility with S3 (which has no concept of them)

Add a path. If *path* hasn't been added before, assign it a name. If *path* is a URI don't add it; just return the URI.

Parse the final output of this MRJob (as a stream of byte chunks) into a stream of ``(key, value)``.

create a dummy job, and call make_runner() on it. Use this in a with block: with self.make_runner() as runner: ...

Update self.mock_s3_fs with a map from bucket name to key name to data.

Check that the param *name* is found in *params*, and if *type* is set, validate that it has the proper type. *type* may also be a tuple (mul

If we're building an inline or Spark runner, include mrjob_cls in kwargs.

Take in data as a sequence of bytes, and yield it, one line at a time. Only breaks lines on ``\\n`` (not ``\\r``), and does not add a trailin

Get the URI for the given path. If *path* is a URI, just return it.

Function to create options which both the job runner and the job itself respect (we use this for protocols, for example). Use it like

Get the ID of the cluster our job is running on, or ``None``.

r"""Return True if *uri* is a URI and contains ``://`` (we only care about URIs that can describe files)

Create a :py:mod:`boto3` EMR client. :return: a :py:class:`botocore.client.EMR` wrapped in a :py:class:`mrjob.retry.RetryWrap

Run a job with the given option dictionary, and check for for instance, number, and optional bid price for each instance role. Specif

Build an instance group request with the given list of EBS device configs. Optionally turn on EBS optimization and specify a different

Build a list of extra args to the spark-submit binary for the given spark or spark_script step.

build a command line that works in a shell.

Launch a job but don't wait for it to finish.

Parse a setup/bootstrap command, finding and pulling out Hadoop Distributed Cache-style paths ("hash paths"). :param string cmd: shell comman

Raise ParamValidationError if *value* isn't an instance of *type*.

Create a tmp directory on the local filesystem that will be cleaned up by self.cleanup()

Get a map from path to URI for all paths that were added, so we can figure out which files we need to upload.

Get a ``datetime`` that's compatible with :py:mod:`boto3`. These are always UTC time, with time zone ``dateutil.tz.tzutc()``.

Return a dictionary representation of this step. See :ref:`steps-format` for examples.

Yield the contents of the file at *path* as a series of ``bytes``, not necessarily respecting line boundaries.

Like yaml.safe_load(), but everything with a !clear tag before it will be wrapped in ClearedValue().

Get the name for a path previously added to this :py:class:`WorkingDirManager`, assigning one as needed. This is primarily for gettin

Come up with a unique name for *path*. :param names_taken: a dictionary or set of names not to use. :param proposed_name: name to use if it i

Return the last value in *values* that is not ``None``. The default combiner; good for simple values (booleans, strings, numbers).

Recursively resolve :py:class:`ClearedValue` wrappers so that ``ClearedValue(...)`` can only wrap values in dicts (and in the top-level value

Returns ``self._steps``.

Create the given dir and its subdirs (if they don't already exist). On cloud filesystems (e.g. S3), also create the corresponding buck

Get a map from name (in the setup directory) to path for all known files/archives, so we can build :option:`-file` and :option:`-archi

Run a MRJob and return its output as a dictionary.

Convert ``bytes`` to unicode. Use this if you need to ``print()`` or log bytes of an unknown encoding, or to parse strings out of bytes of un

Return the URI of where to upload *path* so it can appear in the working dir as *name*, or ``None`` if it doesn't need to be uploaded.

Search for the hadoop streaming jar. See :py:meth:`_hadoop_streaming_jar_dirs` for where we search.

Write out configuration options to a file. Useful if you don't want to bother to figure out YAML. *conf* should look something like this:

Return ``True`` if version >= *cmp_version_str*.

Combine zero or more dictionaries. Values from dicts later in the list take precedence over values earlier in the list. If you pass in ``None

Helper for run_* methods

Attempt to lock the given pooled cluster using EMR tags. You may optionally include *cluster* (a cluster description) and *when_cluster_descr

Yield results from a paginatable API client call. *what* is the name of the field the holds the list of items in each page (e.g. ``'InstanceG

Build a list of extra arguments to the hadoop binary. This handles *cmdenv*, *hadoop_extra_args*, *hadoop_input_format*, *hadoop_outp

Add a filesystem. :param fs: a :py:class:~mrjob.fs.base.Filesystem to forward calls to. :param name string: Name of this filesystem.

Find a cluster that can host this runner. Prefer clusters with more compute units. Break ties by choosing cluster with longest idle time.

Get the version of the AMI that our cluster is running, or ``None``.

Increment a counter in Hadoop streaming by printing to stderr. :type group: str :param group: counter group :type counter: st

Like eval, but with nearly everything in the environment blanked out, so that it's difficult to cause mischief. *globals* and *locals* are op

Check if the requested image version is greater than or equal to *version*. If the *release_label* opt is set, look at that instead.

Get steps for the runner's cluster as a list (in forward order)

Find the directory containing the job output. If the job hasn't run yet, returns None

Allows you to look up something by version (e.g. which jobconf variable to use, specifying only the versions where that value changed. *versi

Assign a path for the archive of *dir_path* but don't actually create anything.

Run a job with the given option dictionary, and check for for instance, number, and optional bid price for each instance role. Specif

cat all files matching **path_glob**, decompressing if necessary This yields bytes, which don't necessarily correspond to lines (see

Find the hadoop binary. A list: binary followed by arguments.

Get the number of steps (calls :py:meth:`get_steps`).

If this is false, we shouldn't attempt to list or cat logs.

Get the total size of files matching ``path_glob`` Corresponds roughly to: ``hadoop fs -du path_glob``