Get a set of all paths tracked by this WorkingDirManager.

Return string to log/print explaining the given error.

Call ListClusters, and collect cluster IDs relevant to pooling. Optionally, only list clusters created after *created_after*. Return

Attempt to explain any error in task stderr, be it a Python exception or a problem with a setup command (see #1203). Looks for '+ ' followed

Pick most recent error from a dictionary possibly containing step, history, and task interpretations. Returns None if there are no errors.

Add a path as either a file or an archive, optionally assigning it a name. :param type: either ``'archive'`` or ``'file'`` :p

Assert that we can encode and decode the given key and value, and get the same key and value we started with.

Resolve ``~`` (home dir) and environment variables in *path*. If *path* is ``None``, return ``None``.

A wrapper for :py:func:`urlparse.urlparse` that splits the fragment correctly in all URIs, not just Web-related ones. This behavior was fixed

Helper for _check_cluster_spark_support().

Create the setup wrapper script, and write it into our local temp directory (by default, to a file named setup-wrapper.sh). This will

Is this the path/URI of a task log? (Including Spark) If so, return a dictionary containing application_id and container_id (on YARN) or atte

URI to use as output for the given step. This is either an intermediate dir (see :py:meth:`intermediate_output_uri`) or ``self._output

Return a :py:class:`google.cloud.storage.bucket.Bucket` Raises an exception if the bucket does not exist.

Create a :py:mod:`boto3` S3 client, wrapped in a :py:class:`mrjob.retry.RetryWrapper` :param region: region to use to choose S3 endpo

Recursively delete the given file/directory, if it exists Corresponds roughly to: ``hadoop fs -rm -R path_glob``

Parse lines from a hadoop log into log4j records. Yield dictionaries with the following keys: caller_location -- e.g. 'YarnClientImpl.java:su

A list of URIs to use as input for the given step. For all except the first step, this list will have a single item (a directory).

Does this runner use Spark, based on steps, bootstrap actions, and EMR applications? If so, we'll need more memory.

Load a list of dictionaries representing the options in a given mrjob.conf for a specific runner, resolving includes. Returns ``[(path, values

Switches for all command line tools

add switches for the given runner opts to the given ArgumentParser, alphabetically by destination. If *opt_names* is None, include all runner

Does the given path/URI exist? Corresponds roughly to: ``hadoop fs -test -e path_glob``

Get the (:py:mod:`boto3`) bucket, connecting through the appropriate endpoint.

Get the value of a jobconf variable from the runtime environment. For example, a :py:class:`~mrjob.job.MRJob` could use ``jobconf_from_env('m

Parse hash paths from old setup/bootstrap options. This is similar to parsing hash paths out of shell commands (see :py:func:`parse_setup_cmd

Set up logging when running from the command line. This is also used by the various command-line utilities. :param bool quiet: If tru

Add files needed for running the job (setup and input) to self._upload_mgr.

Given a list of (read_protocol_class, write_protocol_class) tuples for *each substep*, assert that the given _steps_desc() output for each

Get the jobconf dictionary, optionally including step-specific jobconf info. Also translate jobconfs to the current Hadoop version, i

Pick probable cause of failure (only call this if job fails).

Concatenate the given sequences into a list. Ignore ``None`` values. Generally this is used for a list of commands we want to run; the "defau

Create a bucket on GCS, optionally setting location constraint. and time-to-live.

return the file extension, including the ``.`` >>> file_ext('foo.tar.gz') '.tar.gz' >>> file_ext('.emacs') '' >>> file_ext('.mr

If path isn't an ``hdfs://`` URL, turn it into one.

Get the value of a jobconf variable from the given dictionary. :param dict jobconf: jobconf dictionary :param string name: name of the jobcon

Parse counters and status messages out of MRJob output. :param stderr: a filehandle, a list of lines (bytes), or bytes :param counters: Count

If *path_or_uri* is not a URI already, convert it to a ``file:///`` URI.

Translate *variable* to Hadoop version *version*. If it's not a variable we recognize, leave as-is.

Basically, is this Hadoop 2? This also handles versions in the zero series (0.23+) where YARN originated.

Add files necesary for the master node setup script to self._master_node_setup_mgr() and self._upload_mgr(). Create the master node s

Convert a cluster to a full summary for use in creating a report, including billing/usage information. :param cluster: a :py:mod:`boto3` clus

Get the mock cluster with the given ID, or raise an InvalidRequestException.

Are any of our steps Spark steps? (e.g. spark, spark_jar, spark_script) Generally used to determine if we need to install Spark on a

Parse *lines* from the ``hadoop jar`` command's stderr (lines can be either bytes or unicode). Works like :py:func:`_parse_step_syslog` (same

Parse a Spark log, looking for errors and application_id

Everything in the *py_files* opt, plus a .zip of the mrjob library if needed.

A list of args to the spark script/jar/MRJob, used by _args_for_spark_step(). *last_step_num* is only used by the Spark runner, where

Make a runner for a two step job and launch it.

A random 16-digit hex string.

Yield items from *item* in order, skipping duplicates.

Add files needed by the bootstrap script to self._upload_mgr. Create the master bootstrap script if necessary. persistent -- set by

Yield lines from the given log. Log errors rather than raising them.

Handle the common case, where a method operates on a single path.

Convert a map from group -> counter name -> amount to a message similar to that printed by the Hadoop binary, with no trailing newline.

Scan node manager log to find failed container IDs of failed tasks, and then scan the corresponding stderr and syslogs.

Call _ls_logs() with self.mock_fs and self.mock_matcher, and return a list.

Yield paths/uris of all job history files in the given directories, optionally filtering by *job_id*.

Collect useful info from a pre-YARN history file. See :py:func:`_parse_yarn_history_log` for return format.

Add information about a mock EC2 Image (AMI) to be returned by mock :py:meth:`~tests.mock_boto3.ec2.MockEC2Client.describe_images`. T

Can we handle this path at all?

Take zero or more commands to run on the command line, and return the last one that is not ``None``. Each command should either be a list cont

Look for :file:`mrjob.conf`, and return its path. Places we look: - The location specified by :envvar:`MRJOB_CONF` - :file:`~/.mrjob.conf`

Map an HDFS URI to a path on the filesystem.