Does the given actual EBS volume satisfy the given request?

Does the given list of actual EBS volumes satisfy the given request? Just compare them one by one (we want each actual device to be bigger/fa

Set up environment variables for a subprocess (mapper, etc.) This combines, in decreasing order of priority: * environment variables

Helper function for the *upload_args methods. The names of archives to pass to the ``--files`` switch of ``spark-submit``, since we ca

Helper function for the *upload_args methods. The names of all arguments to ``-files`` (or ``--files`` on Spark). If *always_use_hash

Filter out clusters that have tags matching any specified in exclude_strings. :param cluster_summaries: a list of :py:mod:`boto3` cluster sum

Look for the hadoop binary in any plausible place. If all else fails, return ``['hadoop']``.

Attempt to find the spark binary. Returns a list of arguments. Defaults to ``['spark-submit']``. Re-define this in your subclass if y

Fix a cleanup option, or raise ValueError.

Return copy of *c* with *Properties* is always set (defaults to {}) and with *Configurations* is not set if empty. Convert all values to strin

Take an options dictionary, and either return a sanitized version of it, or raise an exception. *source* is either a string describin

Fill in cloud_tmp_dir and cloud_log_dir (in self._opts) if they aren't already set. Helper for __init__.

Return either None, a string, or a list with at least two items.

Ensure that exactly one of region and zone is set.

Make sure the specification for the given instance type is as good or better than the requested spec. Specs must have the same weight, but "b

Return string to log/print explaining the given error.

Format a timedelta for use in a columnar format. This just tweaks stuff like ``'3 days, 9:00:00'`` to line up with ``'3 days, 10:00:00'``

Returns a map from lowercase app name to version for our cluster. For apps other than Hadoop, this only works for AMI 4.x and later.

Return the collection type of the cluster (either ``'INSTANCE_FLEET'`` or ``'INSTANCE_GROUP'``).

Efficiently fetch steps for a particular mrjob run from the EMR API. :param emr_client: a boto3 EMR client. See :py:meth:`

Extract the options for the given runner class from *options*.

Extract the step from the runner options.

Return interpretation of the step log. Either implement this, or fill ``'step'`` yourself (e.g. from Hadoop binary's output.

Figure out the output dir for a step by parsing step.args and looking for an -output argument.

Helper for _fix_tmpdir

Helper for _split_records().

Group streaming steps together.

Escape args of a command line in a way that Hadoop can process them.

Escape a single command argument in a way that Hadoop can process it.

Given a path to the hadoop binary, return the path of the implied hadoop home, or None if we don't know. Don't return the parent directory of

Yield all possible places to look for the Hadoop streaming jar. May yield duplicates.

Convert a string like ``root-device-type`` to ``RootDeviceType``

Does the actual instance group definition satisfy the bid price (or lack thereof) of the requested instance group?

Does the actual instance group satisfy the memory requirements of the requested instance group?

Does the *actual* list of instance groups satisfy the *requested* one?

Does the list of actual instance groups satisfy the CPU requirements of the requested instance group?

What's the bid price for the given role (if any)?

Common code for :py:func:` :py:func:`_instance_groups_satisfy_fleets` and :py:func:`_instance_groups_satisfy`.

Do the actual instance groups from a cluster satisfy the requested ones, for the purpose of pooling?

Do instances of the given role run tasks? True for non-master instances and sole master instance.

What instance type should we use for the given role? (one of 'MASTER', 'CORE', 'TASK')

Validate InstanceTypeConfigs from fleet request, and convert to InstanceTypeSpecifications (from ListInstanceFleets).

Redefine this in your subclass if the given path needs to be translated to a URI when running spark (e.g. on EMR).

Incrementally update *step_interpretation* (a dict) with information from new lines read from Hadoop job driver output on Dataproc.

Look for errors in task syslog/stderr. If *partial* is true (the default), stop when we find the first error that includes a *task_error*.

Fetch task syslogs and stderr, and add 'task' to interpretation.

Return a function that runs the given mapper/reducer. This needs to be pickleable if tasks are going to be invoked through multiproces

Does the given step type indicate that it uses Spark and Python?

Returns a map from ``'hadoop_job'`` to a dict representing a Hadoop job that runs a JAR

Keyword arguments to the runner class that can be specified by the job/launcher itself.

Not actually used to set up the SSH tunnel, used to run curl over SSH to fetch from the job tracker directly.

The command used to create a :py:class:`subprocess.Popen` to run the SSH tunnel. You usually don't need to redefine this.

Helper for :py:func:`load_opts_from_mrjob_conf` for recursive use. This doesn't expand or default *conf_path*.

Log why the job failed.

return a map from name to value into a log filter query that requires each name to equal the given value.

Helper method for _ls_task_logs() and _ls_spark_task_logs().

Log a log4j message at the appropriate logging level

Find all stderr from bootstrap actions in the given dir. Sort so the most recent one comes first, using node ID as a tiebreaker. (In practice

Yield matching files, optionally filtering by *job_id*. Yields dicts with the keys: job_id: job_id in path (must match *job_id* if set) p

Make an arg parser that's used only for printing basic help. This prints help very similar to spark-submit itself. Runner args are not includ

Come up with a useful unique ID for this job. Optionally, you can specify a custom label or owner (otherwise we use :py:meth:`_label`

Return a list of ``(glob, cmd)``, where *glob* matches a path or URI to download, and download command is a command to download it (e.

write the part of the manifest setup script after setup, that downloads the input file, runs the script, and then deletes the file.

Return a list of ``(ext, cmd)`` where ``ext`` is a file extension (e.g. ``gz``) and ``cmd`` is a command to uncompress it (e.g. ``gunz