Yield history log matches, logging a message for each one.

Yield step log matches, logging a message for each one.

Return the contents of a tag used to lock a cluster. *expiry* is the unix timestamp for when the lock is no longer valid

Return arguments to add to every invocation of MRJob. :type local: boolean :param local: if this is True, use files' local paths rath

How many of the given instance type do we want?

Simulate parsing by *parser*, return a list of tuples of (dest, option_string, args). If *args* contains unknown args or is otherwise malform

Pulls errors, counters, IDs, etc. from log4j records emitted by Hadoop. This powers :py:func:`_parse_step_syslog` and :py:func:`_interpre

Parse all stderr files from the given task (if any).

Un-escape string from a pre-YARN history file.

Given an :py:class:`EMRJobRunner`, run the command specified by *cmd_args* on all nodes in the cluster and save the stdout and stderr of each

Run the spark submit binary in a subprocess, using a PTY if possible :param spark_submit_args: spark-submit binary and arguments, as as list

Returns a no-args function that runs one mapper, reducer, or combiner. This sets up everything the task needs to run, then passes it

Get the runner subclass corresponding to the given alias (importing code only as needed).

Return a (Bourne) shell script that runs the setup commands and then executes whatever is passed to it (this will be our mapper/reducer),

Simulate progress on the given cluster. This is automatically run when we call :py:meth:`describe_step`, and, when the cluster is ``TE

Copy SSH keys if necessary, then launch the given command over SSH and return a Popen.

Run the given SSH command, and raise an IOError if it fails. Return ``(stdout, stderr)`` Use this for commands with a bounded amount

Does this step run on Spark? (This is re-defined in the Spark runner to include streaming steps, and used by mrjob.logs.mixin)

Get lists of directories to look for the step log in.

Convert e.g. ``'task_201601081945_0005_m_000005'`` or ``'application_201601081945_0005'`` to to ``'job_201601081945_0005'``.

Helper for :py:meth:`archives`, :py:meth:`dirs`, and :py:meth:`files`

Sleep for a little while, to give S3 a chance to sync up.

Wait for every step of the job to complete, one by one.

Pick the protocol classes to use for reading and writing for the given step. Returns a tuple of ``(read_lines, write_line)``

Write a setup script and return its path.

Convert a zone (like us-west1-b) to the corresponding region (like us-west1).

Write the data to a temp file, and then upload it to (mock) S3, checking that the data successfully uploaded.

We can handle any path handled by any (non-disabled) filesystem.

Returns the last value in *paths* that is not ``None``. Resolve ``~`` (home dir) and environment variables.

Get a list of the names of all buckets owned by this user on S3.

Invoke the hadoop executable to determine its version

Fetch the steps submitted by this runner from the EMR API. .. deprecated:: 0.7.4 .. versionadded:: 0.6.1

Instance of the protocol to use to communicate between steps. Default behavior is to return an instance of :py:attr:`INTERNAL_PROTOCOL

Make a runner based on command-line arguments, so we can launch this job on EMR, on Hadoop, or locally. :rtype: :py:class:`mrjob.runn

Make a mock RDD that returns itself.

Create a directory at ``path`` relative to the temp directory for ``host``, where ``path`` is a POSIX path

Use RawProtocol to read output from the jar.

Upload a file on the local filesystem (*src*) to *path*. Like with :py:func:`shutil.copyfile`, *path* should be the full path of the n

Generate fake output on GCS for the given Dataproc runner.

Run the reducer for the given step. :type step_num: int :param step_num: which step to run (0-indexed) Called from :py:meth:

Context manager that saves the current working directory, and chdir's back to it after execution.

disable all logging handlers

Convert *readable*, which is any object supporting ``read()`` (e.g. fileobjs) to a stream of non-empty ``bytes``. If *readable* has an ``__it

Return a dict with the *extra_cluster_params* opt patched into *params*, and ``None`` values removed.

If *d* has *task_id* or *application_id* but not *job_id*, add it. (We don't infer application_id from job_id because application_id only

Add instance groups from *InstanceGroups* to the mock cluster *cluster*.

Helper for :py:func:`_igs_satisfy_request`. Add requests for *missing_roles* to *role_to_ig* so that we have a better chance of matching the c