Helper for _parse_task_syslog(); takes log4j records rather than lines

Utility for logging messages

Fetch progress by calling :py:func:`urlopen` on our ssh tunnel, or return ``None``.

Fetch progress by running :command:`curl` over SSH, or return ``None``

Does the given role data structure have the given policy document and the given policy ARN attached? (Roles can have up to two policy ARNs at

Run our job's combiner, and group lines with the same key together. :param combiner_job: an instance of our job, instantiated to be the mapper

Run multiple no-args functions, possibly in parallel using :py:mod:`multiprocessing` (if so all *funcs* must be pickleable). By defau

Run our job's combiner, and group lines with the same key together. :param reducer_job: an instance of our job, instantiated to be the mapper

Parse command line arguments into arguments for :py:class:`EMRJobRunner`

Call this whenever you think it is possible to SSH to your cluster. This sets :py:attr:`_ssh_proc`. Does nothing if :mrjob-opt:`ssh_tunnel`

A list of lines to put at the very start of any sh script (e.g. ``set -e`` when ``sh -e`` wont work, see #1549)

Simulate Hadoop's shuffle-and-sort step, so that data will be in the format the reducer expects. :param rdd: an RDD containing lines represen

Returns a function that sorts lines from one or more input paths into a new file. Takes the arguments *input_path* and *output_path*.

Given a stream of records (bytestrings, usually lines), yield groups of records (as generators such that the total number of bytes in each group

Close file handles and do error handling on a ``Popen`` who we've read stdout from but done nothing else.

Redefine this in your subclass. Should return a dict with the following keys: *localhost*: once we SSH in, is the web interface?

Options to SSH related to setting up a tunnel (rather than SSHing in). Helper for :py:meth:`_ssh_tunnel_args`.

Get the hostnames of all core and task nodes, that are currently running, so we can SSH to them through the master nodes and read thei

Return a list of lines (without trailing newlines) containing the shell script shebang and pre-commands.

Number that indicates this is the nth task to be launched on the cluster. The second number in any ID. For example, in container_156701349369

Does this step involve running Python on Spark? (This is re-defined in the Spark runner to include streaming steps, and used by mrjob

Return a step data structures to pass to ``boto3``

Set self._image_version and self._hadoop_version.

Stream a single directory on S3 containing the relevant bootstrap stderr. Optionally, use *action_num* and *node_id* to narrow it down

Combine many maps from group to counter to amount.

Symlink from *dest* to *path*, using relative paths if possible. If symlinks aren't available, copy path to dest instead.

When did something last happen with the given cluster? Things we look at: * cluster's ``CreationDateTime`` (always set) * cluster's ``Re

cluster timestamp. the first number in any ID. For example, in container_1567013493699_0003_01_000002, it's 1567013493699.

Look up the unarchive command to use with the given file extension, or raise KeyError if there is no matching command.

Sleep for a little while, to give FS a chance to sync up.

Used to split input for the given mapper/reducer. Yields writeable fileobjs for input splits (check their *name* attribute to get the

Add counters for a subsequent run of the hadoop jar command (streaming or otherwise), as a map from group name to counter name to amount.

Update *mock_s3_fs* with a map from bucket name to key name to data. :param age: a timedelta :param location string: the bucket's location co

call this for tests that are going to invoke a subprocess that needs to find mrjob. (Merely using the local runner won't require this

Same as :py:func:`combine_envs`, except that paths are combined using the local path separator (e.g ``;`` on Windows rather than ``:``).

Runs :py:meth:`combiner_init`, :py:meth:`combiner`, and :py:meth:`combiner_final` for one reduce task in one step. Takes in a

Dump a wrapper script to the given file object that runs this python script.

Rot13-encode the given string.

Return the hadoop binary, searching for it if need be.

Find the path of the hadoop streaming jar, or None if not found.

Return the version number of the Hadoop environment as a string if Hadoop is being used or simulated. Return None if not applicable.

Get a list for each invocation of hadoop, each containing a list of arguments (not including the hadoop binary's path).

Optional Hadoop ``InputFormat`` class to parse input for the first step of the job. Normally, setting :py:attr:`HADOOP_INPUT_FORMAT`

Optional Hadoop ``OutputFormat`` class to write output for the last step of the job. Normally, setting :py:attr:`HADOOP_OUTPUT_FORMAT

Instance of the protocol to use to convert input lines to Python objects. Default behavior is to return an instance of :py:attr:`INPUT

Helper function to call command and subcommands of the hadoop binary. Basically, combines prefix and cmd to make a function name, and calls i

Return True if *uri* can be parsed into an S3 URI, False otherwise.

Return the current time as YYmmdd.HH:MM:SS.uuuuuu. Use this to ensure that mock data added stays in alphabetical order.

List buckets on GCS.

Set up a null handler for the given stream, to suppress "no handlers could be found" warnings.

Set up logging. :type name: str :param name: name of the logger, or ``None`` for the root logger :type stream: file object :param str

Runs :py:meth:`mapper_init`, :py:meth:`mapper`/:py:meth:`mapper_raw`, and :py:meth:`mapper_final` for one map task in one step.

Main func should take the arguments (stdin, stdout, stderr, argv, environ_dict).