Release our lock on the given pooled cluster. Only do this if you know the cluster is currently running steps (so other jobs won't try to join

Parse *bootstrap_actions* option into dictionaries with keys *path*, *args*, adding Spark bootstrap action if needed. (This doesn't h

Should we bootstrap mrjob?

return either '<path>' or 'line N of <path>' or 'lines M-N of <path>'.

If *host_or_uri* is non-empty and isn't a URI, prepend ``'https://'``. Otherwise, pass through as-is.

Get the paths to input files, dumping STDIN to a local file if need be.

Wrapper to get at the underlying function belonging to a method. Python 2 is slightly different because classes have "unbound methods" which

Create an empty cluster on Dataproc, and set self._cluster_id to its ID.

Get bootstrap action for the runner's cluster as a list.

Yield task log matches.

Where to place files used by the master node setup script.

Get the internal ("private") address of the master node, so we can direct our SSH tunnel to it.

Match path of a step syslog, optionally filtering by *step_id*. If there is a match, return a dict with the keys *step_id* and *timestamp* (a

Return (job_key, expiry) or raise ValueError Raises TypeError if *lock* is not a string.

Pull (map_percent, reduce_percent) from running job from job tracker HTML as floats, or return (None, None). This assumes at most one running

Set up files needed for the job.

Runner class as indicated by ``--runner``. Defaults to ``'inline'``.

Delete all files older than *time_old* in *path*. If *dry_run* is true, then just log the files that need to be deleted without actually dele

Sort the given list/sequence of dicts in forward order by task/container ID, with most recent attempts first. This is used for finding errors on

Use the local-cluster master, which simulates a Spark cluster.

The path of the spark script or JAR, used by _args_for_spark_step().

Stream log dirs for any kind of log. Our general strategy is first, if SSH is enabled, to SSH into the master node (and possibly work

Where to output data for the given task. Usually this is just a file named "output" in the task's directory, but if it's the last tas

Write text of a setup script, input manifest, etc. to the given file. By default, this writes binary data. Redefine :py:meth:`write_l

Simulate successful creation of new metadata tags for the specified resource id.

Concatenate the given sequences into a list. Ignore None values. Resolve ``~`` (home dir) and environment variables, and expand globs that ref

Invoke the hadoop executable to determine its version

return the gzip-compressed version of the given bytes.

Shortcut for automatically loading mrjob.conf from one of the predefined locations and returning the de-YAMLed object

Make an EMRJobRunner that is ready to try to find a pool to join

Use this to make a mock stdout/stderr (e.g. for a method that calls print()). In Python 2, this is just a StringIO. In Python 3, this is

Instance of the protocol to use to convert Python objects to output lines. Default behavior is to return an instance of :py:attr:`OUTP

Convert a POSIX path to the current system's format and prepend the tmp directory the host's files are in

Set the job status in hadoop streaming by printing to stderr. This is also a good way of doing a keepalive for a job that goes a long

Add steps (from _steps_to_submit()) to our cluster and append their IDs to self._step_ids

Utility for :py:meth:`MRJobRunner._combine_opts()`: if multiple configs specify conflicting opts, blank them out in all but the last config (s

Helper for __init__

Upload working dir files to the working dir mirror, if necessary. This does not handle archives, which we always rename with hash pat

True if spark-submit's --archives doesn't work on the given Spark master, which means we'll need to emulate archives in setup scripts.

Convert environment dictionary to strings (Python 2.7 on Windows doesn't allow unicode).

Get a single step (calls :py:meth:`_get_steps`).

The arguments that come after ``hadoop jar <streaming jar path>`` when running a Hadoop streaming job.

Call method named *name* on the first (non-disabled) filesystem that says it can handle every path in *paths_to_handle*. If it raises

A directory for intermediate output for the given step number.

Like :py:meth:`_interpolate_step_args` except it also replaces `~mrjob.step.GENERIC_ARGS` with :py:meth:`_hadoop_generic_args_for_step

Return ``True`` if *cluster_summary* is currently bootstrapping.

Return True if the given cluster is done running.

Send SIGKILL to SSH tunnel, if it's running.

Helper method for :py:meth:`ls`; yields tuples of ``(uri, blob)`` where *blob* is the corresponding :py:class:`google.cloud.storage.bl