A list of hard-coded commands to run at the beginning of the bootstrap script. Currently used by dataproc to cd into a tmp dir.

Helper for __init__

Raise :py:class:`NotImplementedError` if the given substep (e.g. ``'mapper'``) runs subprocesses.

make *path* user readable and executable. If *recursive* is true, make *path* and everything inside it executable.

Keyword args for creating resources or clients.

Does *cluster* have any steps in the ``PENDING`` state?

Extra info added to the cluster name, for pooling.

Extract fields such as creation time, owner, etc. from the cluster. :param cluster: a :py:mod:`boto3` cluster data structure :param now: the

Helper for :py:meth:`combine_pairs` and :py:meth:`reduce_pairs`.

Upload/copy *path* to the appropriate place in the working dir mirror, if necessary. We don't track whether something has already bee

Create an empty cluster on EMR, and return the ID of that job. If the ``tags`` option is set, also tags the cluster (which is

Copy working directory files into a shared directory, simulating the way Hadoop's Distributed Cache works on nodes.

Helper for :py:meth:`_add_bootstrap_files_for_upload`. Create the master bootstrap script and write it into our local temp directory.

Create a new role with a random name starting with ``mrjob-`` that has the given policy document and the given policy ARN attached. (Roles ca

Takes the value of the *emr_configurations* opt, and ensures that later configs overwrite earlier ones with the same Classification. Addition

Default instance type if not set by the user.

Helper function for :py:func:`_run_combiner` and :py:func:`_shuffle_and_sort`. Given an RDD containing (key, [line1, line2, ...]), discard *k

Special-case the "library" registry which is implied on Docker Hub but needs to be specified explicitly on EMR.

Encode a single key/value, and return it.

Does *actual* have EBS volumes that satisfy *request*. *actual* is either an instance group from ``ListInstanceGroups`` or an instance fleet

Yield lines from a PTY, gracefully handling an ``IOError`` with ``errno == EIO`` as end-of-file.

Return the path of a single conf file. If *conf_path* is ``False``, return ``None``, and if it's ``None``, return :py:func:`find_mrjob_conf`.

A list of additional local files expected, as tuples of (path, name). Hook for dealing with cat.py in local mode.

Helper method: Extract *action_num* and *node_id* from the given regex match. Convert *action_num* to a 0-indexed integer.

Extract all errors from *log_interpretation*, in no particular order.

Convert Avro-Json counter data structure to our group -> counter -> amount format. This deals gracefully with unexpected data structures.

Identify jobs that have been running or pending for a long time. :param clusters: a list of :py:mod:`boto3` cluster summary data structures :

Fix a single option, returning its correct value or raising an exception. This is not called for options that are ``None``. This curr

Look up the given bucket's location constraint and translate it to a region name.

Get a list of complete job driver output lines that are new since the last time we checked.

Yield all possible hadoop directories (used for streaming jar and logs). May yield duplicates

Implements fs -du and fs -dus.

Helper for hadoop_fs_ls() and hadoop_fs_lsr().

Helper for hadoop_fs_rm() and hadoop_fs_rmr().

Arguments like -D and -libjars that apply to every Hadoop subcommand.

Are any of our steps Hadoop Streaming steps?

Do any of our steps involve running Python on Spark? Includes spark and spark_script types, but not spark_jar. Generally used to tell

Are any of our steps "streaming" steps that would normally run on Hadoop Streaming?

Which instance groups do we want to request? Returns the value of the ``InstanceGroups`` parameter passed to the EMR API.

Replace :py:data:`~mrjob.step.INPUT` and :py:data:`~mrjob.step.OUTPUT` in arguments to a jar or Spark step.

Extract errors from bootstrap stderr. If *partial* is true, stop when we find the first match. (In practice, we usually target a single file

Extract information from step stderr (see :py:func:`~mrjob.logs.task._parse_task_stderr()`), which may be split into several chunks by timesta

Extract information from step syslog (see :py:func:`_parse_step_log()`), which may be split into several chunks by timestamp

Extract counters and errors from history log. Matches is a list of dicts with the keys *job_id* and *yarn* (see :py:func:`_ls_history_logs()`

Fetch history log and add 'history' to log_interpretation.

Add *step* to the log interpretation, if it's not already there.

Is this the record containing counters? (HadoopJobRunner doesn't want to log this because it's very long and it has its own way of displaying

Return true if the given step is currently running.

A unique module name to use with the MRJob script.

The host of the job tracker/resource manager, from the master node.

Return *label* opt, or if not set, the name of the file containing the MRJob, minus extension, or if none, ``'no_script'``

Create an empty cluster on EMR, and set self._cluster_id to its ID.

Paths or URIs of libjars, from Hadoop/Spark's point of view. Override this for non-local libjars (e.g. on EMR).

Helper for list_steps() and describe_step().

Decode a single key/value, and return it.

Log the master node's public DNS, if we haven't already

Convert log entries from a single log file to log4j format, tracking line number. See :py:meth:`mrjob.logs.log4j._parse_hadoop_log4j_records`

Log ``' <line>'``. *line* should be a string. Optionally specify a logging level (default is logging.INFO).

Tunnel to the job tracker/resource manager and log the progress of the current step. (This takes no arguments; we just assume the mos

Do ``fs.exists(path)``, and return ``None`` if it raises ``IOError``

_ls_bootstrap_stderr_logs(), with logging for each log we parse.

Yield matching step logs, optionally filtering by *step_id*. Yields dicts with the keys: path: path/URI of step file step_id: step_id in

Yield matching step logs, optionally filtering by *step_id*. Yields dicts with the keys: path: path/URI of step file step_id: step_id in

Yield matching Spark logs, optionally filtering by application_id or job_id. This will yield matches for stderr logs only. stderr logs wi

Yield step log matches, logging a message for each one.

Yield matching logs, optionally filtering by application_id or job_id. This will yield matches for stderr logs first, followed by syslogs. st

Helper for _ls_task_logs() and _ls_spark_task_logs(). *syslog_type* is the type of the log to pair with stderr logs. This is actually a bit

Helper for :py:func:`_match_emr_step_syslog_path` and :py:func:`_match_emr_step_stderr_path`

Return *owner* opt (which defaults to :py:func:`getpass.getuser`), or ``'no_user'`` if not set.