Get the URI of the log directory for this job's cluster.

Assert that we can encode the given key and value, add a trailing tab (which Hadoop sometimes does), and decode it to get the same key

Get the path of mock root of HDFS. Creates the directory if it doesn't already exist.

make and launch runner, so cluster is created and files are uploaded.

Create a :py:mod:`boto3` IAM client. :return: a :py:class:`botocore.client.IAM` wrapped in a :py:class:`mrjob.retry.RetryWrap

Return a copy of BASE_EMR_IMAGE with the given fields added. You can blank out fields by setting them to None.

Create a :py:mod:`boto3` S3 resource, with its client wrapped in a :py:class:`mrjob.retry.RetryWrapper` :param region: region to use

Path of this script. This returns the file containing this class, or ``None`` if there isn't any (e.g. it was defined from the command

Context manager that saves os.environ and loads it back again after execution

A method that by default, just returns the value of :py:attr:`SORT_VALUES`. Mostly exists for the sake of consistency, but you could o

Return the given :py:class:`datetime.timedelta`, without microseconds. Useful for printing :py:class:`datetime.timedelta` objects.

Make an empty file in the given location. Raises an error if a non-zero length file already exists in that location. Correponds to: `

Translates the configuration property name to match those that are accepted in hadoop_version. Prints a warning message if any configuration p

Use this to turn on *sudo* (we do this depending on the AMI version on EMR).

If *d* (a dictionary) has *attempt_id* but not *task_id*, add it. Use this on errors.

The actual arguments used to run the spark-submit command. This handles both all Spark step types (``spark``, ``spark_jar``, and ``sp

If our bootstrap actions failed, parse the stderr to find out why.

Given a reason for cluster termination, check if it's because a bootstrap action terminated with an error. If it is, return a dictionary with

get tags from a cluster as a dict, excluding tags starting with ``__mrjob_``

Find hadoop and (if needed) spark-submit bin up-front, before continuing with the job. (This is just for user-interaction purposes; t

Get the boto3 s3.Object matching the given S3 uri, or return None if that key doesn't exist. uri is an S3 URI: ``s3://foo/bar``

Fetch and interpret the step log.

Does nothing. We can't get the history logs, and we don't need them.

Does the given step type indicate that it uses Spark?

Parse a COUNTERS field from a pre-YARN history file. Returns a map from group to counter to amount.

Pull progress_precent for running job from job tracker HTML, as a float, or return None. This assumes at most one running job (designed for E

Parse the syslog from the ``hadoop jar`` command. Returns a dictionary which potentially contains the following keys: application_id: a stri

Pick counters from our log interpretation, interpreting history logs if need be.

Returns a dictionary mapping environment variable to value, including mapping PYSPARK_PYTHON to self._python_bin()

Look up AMI version, and return a dict with the following keys: name: "job tracker" or "resource manager" path: path to start page of

Treat streaming steps as Spark steps that use Python.

remove the clear tag from the given value.

Part size for uploads, in bytes, or ``None``, from :mrjob-opt:`cloud_part_size_mb`

Does the first step take an input manifest?

If the cluster is already terminating, wait for it to terminate, so that logs will be transferred to S3. Don't print anything unless

Add mock output which will be used by the next fake streaming job that mockhadoop will run. Args: parts -- a list of the contents of part

Assert that if we create an EMRJobRunner with the given keyword args, it'll create a new tmp bucket with the given location constraint

Assert that if we create an DataprocJobRunner with the given keyword args, it'll create a new tmp bucket with the given location const

The master combiner, used to combine dictionaries of options with appropriate sub-combiners. :param combiners: a map from option name to a co

Create a bucket on S3 with a location constraint matching the given region.

Fetch a list of Amazon Linux AMI images that are usable with EMR, with the most recent first. This can take several seconds. :param ec2_clien

Wrap _interpret_emr_bootstrap_stderr(), since fs doesn't matter

Wrap _interpret_history_log(), since fs doesn't matter.

True if this is a mapper, combiner, reducer, or Spark script. This is mostly useful inside :py:meth:`load_args`, to disable loading a

Load a list of dictionaries representing the options in a given list of mrjob config files for a specific runner. Returns ``[(path, values), .

Generate the md5 sum of the file at *path*

Parse a filename like ``some_id-cat1-cat2-not_cat3.txt`` into ``dict(id='some_id', cats=dict(cat1=True, cat2=True, cat3=False))``

Pass the given argument through to the job.

Get the filesystem path that the given host is being faked at

Run the mapper and final mapper action for the given step. :type step_num: int :param step_num: which step to run (0-indexed)

Context manager that saves the current values of `sys.stdin`, `sys.stdout`, and `sys.stderr`, and flushes these filehandles before and after s

Re-define this to make a multi-step job. If you don't re-define this, we'll automatically create a one-step job using any of :py:meth

Get all known variants of the given jobconf variable. Unlike :py:func:`translate_jobconf`, returns a list.

Set all attributes to the corresponding value in *kwargs*, or the default value. Raise :py:class:`TypeError` for unknown arguments or