The directory containing the mrjob package that we've imported.

Optional Hadoop partitioner class to use to determine how mapper output should be sorted and distributed to reducers. By default, ret

Uploads a local file to a specific destination.

Temporarily change the seed of the random number generator.

Map a real path to an hdfs:/// URI.

Runs :py:meth:`reducer_init`, :py:meth:`reducer`, and :py:meth:`reducer_final` for one reduce task in one step. Takes in a se

Simulate successful creation of new metadata tags for the specified resource id.

If any node is a pending_xref, attempt to resolve it. If it cannot be resolved, replace it with its children.

Context manager that saves sys.path and restores it after execution.

Manually set the hadoop binary, as a list of args.

Extract the contents of a tar or zip file at *archive_path* into the directory *dest*. :type archive_path: str :param archive_path: path

Arguments to this constructor may also appear in :file:`mrjob.conf` under ``runners/local``. :py:class:`~mrjob.local.LocalMRJobRunner

Human-readable version of the exception. Note that this 1-indexes *step_num*.

Wrap method f in a retry loop.

Handle BootstrapActions param from run_job_flow(). (there isn't any other way to add bootstrap actions)

Add instance fleets from *InstanceFleets* to the mock cluster *cluster*. This is just a helper for :py:meth:`add_instances`; there is

Handle Instances param from run_job_flow()

Add files needed for running the job (setup and input) to self._upload_mgr.

Add files needed for running the job (setup and input) to self._upload_mgr.

Add switches for a single option (*opt_name*) to the given parser.

Add switches that determine what part of the job a MRJob runs.

Add tags in the dict *tags* to cluster *cluster_id*. Do nothing if *tags* is empty or ``None``

Helper function for the *upload_args methods. The names of all arguments to ``-archives`` (or ``--archives`` on Spark).

Convert e.g. ``'attempt_201601081945_0005_m_000005_0'`` to ``'task_201601081945_0005_m_000005'``

Which attempt this is for a particular task; typically tasks that fail are retried one or two times. This applies to both attempt and containe

Redefine this to return a (possibly empty) list of parsed commands (in the same format as returned by parse_setup_cmd())' to make sure a

Helper method for creating instance groups. For use when creating a cluster using a list of InstanceGroups - role is either 'MASTER', 'CO

:py:meth:`cat_file`, minus decompression.

Issue a warning if our cluster doesn't support Spark. This should only be called if you are going to run one or more Spark steps.

If cluster couldn't start due to missing IAM roles, tell user what to do.

If failure could have been due to a pooled cluster self-terminating, raise _PooledClusterSelfTerminatedException

Raise :py:class:`IOError` if the given input does not exist or is otherwise invalid. Override this to provide custom check behavior.

Check that input exists prior to running the job, if the `check_input_paths` option is true.

Verify the output path does not already exist. This avoids provisioning a cluster only to have Hadoop refuse to launch.

Verify the output path does not already exist. This avoids provisioning a cluster only to have Hadoop refuse to launch.

Check that the given step description is for a MRStep with no input manifest

Raise an exception if the given step is invalid (:py:class:`ValueError`) or not handled by this runner (:py:class:`NotImplementedError

Look at the step definition (*steps*). If it is not supported by the runner, raise :py:class:`NotImplementedError`. If it is not suppo

Cast any dictionary keys in the given JSON object to str. We can assume that x isn't a recursive data structure, and that this is only called

Cleanup any files/directories on cloud storage (e.g. S3) we created while running this job. Should be safe to run this at any time, or

Terminate the cluster if there is one.

Cleanup any files/directories on HDFS we created while running this job. Should be safe to run this at any time, or multiple times.

Stop any jobs that we created that are still running.

Cleanup any files/directories on the local machine we created while running this job. Should be safe to run this at any time, or multiple

Cleanup any log files that are created as a side-effect of the job.

Actual cleanup action to take based on various options

Get the error code for the given ClientError

Do we want to join the cluster with the given description?

Check if the cluster has an instance group/fleet configuration that works as well or better. This either calls ``ListInstanceFleets``

Build kwargs for emr_client.run_job_flow()

r"""Break billing/usage information for a cluster down by job. :param cluster: a :py:mod:`boto3` cluster data structure :param basic_summary:

r"""Aggregate statistics for several clusters into a dictionary. :param clusters: a sequence of dicts with the keys ``cluster``, ``steps``. :

Combine several opt dictionaries into one. *source_and_opt_list* is a list of tuples of *source*, *opts* where *opts* is a dictionary

Combine several opt dictionaries into one. *opt_list* is a list of dictionaries containing validated options Override this if you nee

Command to copy files from the cloud to the local directory.

Command to copy files from the cloud to the local directory (usually via Hadoop). Redefine this as needed; for example, on EMR, we som

Call this to create all dir archives

Create a file with a list of URIs of input files.

Helper for :py:meth:`_add_bootstrap_files_for_upload`. If we need a master node setup script and write it into our local temp directo

The default python command. If local is true, try to use sys.executable. Otherwise use 'python2.7' or 'python3' as appropriate. This

The default sh binary, if :mrjob-opt:`sh_bin` isn't set.

Where to put output for steps other than the last one, if not specified by the *output_dir* constructor keyword. Usually you want this

Infer the trusted docker registry from the docker image.

Get total size of file or files in dir (recursive).