Return a list containing the lines of the master bootstrap script. (without trailing newlines)

Create the contents of the master node setup script as an array of strings. (prepare self._master_node_setup_mgr first)

The API will return an empty Properties list for configurations without properties set, and remove empty sub-configurations

Return value of opt to show in debug printout. Used to obfuscate credentials, etc.

A dictionary mapping opt name to combiner funciton. This won't necessarily include every opt name (we default to :py:func:`~mrjob.conf

Parse the *bootstrap* option with :py:func:`mrjob.setup.parse_setup_cmd()`.

Return a dictionary possibly containing the keys: mrjob_version: version of mrjob that created this cluster pool_hash: hash representing boot

Parse a filter of the form: ``foo = "bar" AND baz.quoted = "\"qux\"" AND ...`` into a map from key to value (This is the in

Helper for _parse_spark_log()

Helper method for _add_extra_cluster_params(). Set *name* in *params* to *value* If *name* has one or more dots in it, recursively set the v

Pick error attempt IDs from step and history logs, so we know which task logs to look at (most relevant first)

Pick the shortest Spark error with a traceback.

Pick a list of ports to try binding our SSH tunnel to. We will try to bind the same port for any given cluster (Issue #67)

A dictionary of information that must be matched exactly to join a pooled cluster (other than mrjob version and pool name). The forma

Print all help for the parser. Unlike similar functions, this needs a parser so that it can include custom options added by a :py:class:`~mrjo

Print help for this job. This will either print runner or basic help. Override to allow other kinds of help.

Print final report. :param stats: a dictionary returned by :py:func:`_clusters_to_stats` :param now: the current UTC time, as a :py:class:`da

Takes in a dictionary of info about a long-running job (see :py:func:`_find_long_running_jobs`), and prints information about it on a single (

A list of additional setup commands to emulate Spark's --py-files option on Hadoop Streaming.

Like ``self._setup``, but prepends commands for archive emulation if needed.

Put a string in double quotes, escaping double quote characters

Read from stdin, or one more files, or directories. Yield one line at time. - Resolve globs (``foo_*.gz``). - Decompress ``.g

Create a list from *items*. If we encounter a :py:class:`ClearedValue`, unwrap it and ignore previous values. Used by ``combine_*()`` functions

Helper for :py:func:`parse_setup_cmd`. Resolve ``~`` (home dir) and environment variables in the given path, and unescape backslashes.

Round up to the next second because that's how EMR bills.

Run the job.

Run our job's mapper. :param make_mrc_job: an instance of our job, instantiated to be the mapper for the step we wish to

Returns a no-args function that runs one mapper, plus the corresponding combiner if there is one.

Run the given step on the RDD and return the transformed RDD.

Run an individual step. You can assume that setup wrapper scripts are created and self._counters has a dictionary for that step already.

Run a Step on Spark. Override this in your subclass. You can assume that setup wrapper scripts are created (if relevant) and that self

Wait for every step of the job to complete, one by one.

Run a Hadoop streaming step on simulated Hadoop.

Options to pass to the EMRJobRunner.

Given the command line options, return the arguments to :py:class:`EMRJobRunner`

Return a mapping of ``self._step_key(step_num, step_type)`` -> (place in sort order of all *script* steps), for the purposes of choosi

Helper for _fix_s3_tmp_and_log_uri_opts

Write setup script content to obtain a file lock, run setup commands in a way that doesn't perturb the script, and then release the lo

Helper for _substep_args() Wrap command in sh -c '...' to allow for pipes, etc. Use *sh_bin* option.

The binary to use to sort input. (On Windows, we go straight to sorting in memory.)

Sort the given list/sequence of dicts containing IDs so that the most recent ones come first (e.g. to find the best error, or the best log fil

Sort lines from *input_paths* and output them into *output_path*. If *sort_values* is true, sort by the entire line; otherwise just sort by e

Jobconf dictionary to enable sorting by value.

Partitioner to use with *sort_values* keyword to the constructor.

Where to find the Spark harness.

Take one or more input paths (which may be compressed) and split it to create the input files for the map tasks. Yields "splits", whi

Split a single, uncompressed file containing sorted input for the reducer into input files for each reducer task. Yield the paths of

Add ``self._ec2_key_pair_file`` to the ssh agent with ``ssh-add``.

Return an ssh command that would run the given command on the given *address*. Address consists of one or most hosts, joined by '!' (

Wait this long after launching the SSH process before checking for failure (default 1 second). You may redefine this.

Helper for :py:meth:`_ssh_tunnel_opts`.

Redefine this in your subclass. You will probably want to call :py:meth:`_ssh_tunnel_opts` somewhere in here. Should return the list

Return something like: ``'mr_your_job Step X of Y'``

Describe our cluster, and cache image_version, hadoop_version, and master_public_dns

List master instance for our cluster, and cache master_private_ip.

Yield lists of directories (usually, URIs) to search for history logs in. Usually, you'll want to add logging messages (e.g.

Yield lists of directories (usually, URIs) to search for task logs in. Usually, you'll want to add logging messages (e.g. 'Se

Returns a map from ``'hadoop_job'`` to a dict representing a hadoop streaming job.

Return a (shallow) copy of the given dict, excluding fields starting with underscore.

In task or attempt IDs, a unique number for the individual task within the map or reduce phase. For example, in task_201601081945_0005_m_

Yield lines from a specific stderr log.