Parse a GCS URI into (bucket, key) >>> parse_gcs_uri("gs://walrus/tmp/") ('walrus', 'tmp/') If ``uri`` is not a GCS URI, raise a ValueEr

Parse an S3 URI into (bucket, key) >>> parse_s3_uri('s3://walrus/tmp/') ('walrus', 'tmp/') If ``uri`` is not an S3 URI, raise a ValueErr

Pick the protocol classes to use for reading and writing for the given step. :type step_num: int :param step_num: which step

Add a fake "Hadoop" binary to its own subdirectory of ``self.tmp_dir``, and set *os.environ[envvar]* to point at it. You can use *dirn

Alphabetize arg parser actions for the sake of nicer help printouts.

Pull the tags from a cluster, as a dict.

Set up files and then launch our job on EMR.

If *path* corresponds to a bootstrap stderr file, return a dict with the keys *action_num* (an 0-indexed int) and *node_id*. Otherwise return

Parse an error out of a syslog file (or a Spark stderr file). Returns a dict, possibly containing the following keys: check_stdout:

Collect useful info from a YARN history file, dealing gracefully with unexpected data structures. This returns a dictionary which may contain

Return *bootstrap_spark* option if set; otherwise return true if our job has Spark steps.

Get lists of directories to look for the task logs in.

Convert a time interval to a map from :py:class:`datetime.date` to the number of seconds within the interval on that date. *start* and *end*

Convert a time interval to a map from hours (represented as :py:class:`datetime.datetime` for the start of the hour) to the number of seconds

Helper for :py:meth:`_interpret_task_logs`

Add a command-line option that sends an external file (e.g. a SQLite DB) to Hadoop:: def configure_args(self): s

Like combine_dicts(), but non-string values are converted to Java-readable string (e.g. True becomes 'true'). Keys whose value is None are bla

Take a *readable* which supports the ``.read()`` method correponding to the given path and returns an iterator that yields chunks of bytes, po

Get the unique key for the job run by this runner. This has the format ``label.owner.date.time.microseconds``

Return the location of the ``spark-submit`` binary, searching for it if necessary.

Run the given hadoop command, raising an exception on non-zero return code. This only works for commands whose output we don't care ab

Return True if *uri* can be parsed into an S3 URI, False otherwise.

Create a directory at ``path`` relative to the temp directory for ``host``, where ``path`` is a POSIX path. Returns the path of the resulting

If there is an upload manager, add input files to it.

Get the address of the master node so we can SSH to it

Returns applications (*applications* option) as a set. Adds in ``Hadoop`` and ``Spark`` as needed.

Get the HTTP status for the given ClientError

Yield all possible places to look for hadoop logs.

Does this runner have "Spark" in its *applications* option?

Does it look like this runner has a spark bootstrap install action set? (Anything ending in "/install-spark" counts.)

Get the ith part of some container ID, e.g. container_1567013493699_0003_01_000002. If *id* is None or too short, return

Log "probable cause of failure" log message.

We don't need all the logs when interpreting Spark steps

Merge errors from one or more lists of errors and then return them, sorted by recency. We allow None in place of an error list.

Parse counters in the indented format output/logged by the Hadoop binary. Takes input as lines (should not include log record stuff) and

Return what percentage *x* is of *total*, or *default* if *total* is zero.

Python binary used for everything other than invoking the job. For running job tasks (e.g. ``--mapper``, ``--spark``), we use :py:met

The sh binary and any arguments, as a list. Override this if, for example, a runner needs different default values depending on circum

Do spark executors have a working directory different from the one *spark-submit* was run in? (True on everything but local.)

Yield lists of directories to look for the history log in.

Python binary used to invoke job with ``--mapper``, ``--reducer``, ``--spark``, etc.

Check if we can find the hadoop binary from *envvar*

A directory to upload files belonging to :py:attr:`_working_dir_mgr`. This will be a subdir of ``self._upload_mgr.prefix`, if it exist

Wrap a given boto3 Client object so that it can retry when throttled.

Combine zero or more dictionaries containing environment variables. Environment variable values may be wrapped in :py:class:`ClearedValue`. E

Get a directory within $MOCK_HADOOP_TMP, creating it if it doesn't already exist. Takes optional *environ* keyword arg (defaults to ``os.envi

Wrapper around shlex.split(), but convert to str if Python version < 2.7.3 when unicode support was added.

Like the UNIX which command: search in *path* for the executable named *cmd*. *path* defaults to :envvar:`PATH`. Returns ``None`` if no such e

Add files needed by the bootstrap script to self._upload_mgr. Create the master bootstrap script if necessary.

Return a (possibly empty) list of parsed commands (in the same format as returned by parse_setup_cmd())

Helper for :py:meth:`archive_dir`

Make a zip of the mrjob library, without .pyc or .pyo files, This will also set ``self._mrjob_zip_path`` and return it. Typically cal

Helper function; set the given attribute to *value* if it's None.

Stream container IDs of failed tasks, in reverse order.

Get state change reason message.

Hook for interpreting step logs. Unlike with most runners, you may call this multiple times and it will continue to parse the step lo

Helper method for :py:meth:`ls`; yields tuples of ``(uri, key)`` where *key* is the corresponding boto3 s3.ObjectSummary.

Return a list matches against log files. Used to implement ``_ls_*_logs()`` functions. This yields dictionaries with ``path`` set to matching

Make a launcher, add a mock runner (``launcher.mock_runner``), and set it up so that ``launcher.make_runner().__enter__()`` returns ``

Match path of a step stderr log, optionally filtering by *step_id*. If there is a match, return a dict with the keys *step_id* and *timestamp

Keyword arguments to runner constructor that can't be set in mrjob.conf. These should match the (named) arguments to :py:meth

r"""Yield records from the given sequence of lines. For example, a line like this: Task TASKID="task_201512311928_0001_m_000003" \ TASK_T

Pick counters from a dictionary possibly containing step and history interpretations.