Yield log4j records from a specific syslog.

In task or attempt IDs, whether this is a map or reduce (not used by Spark). For example, in attempt_201601081945_0005_m_000005_0, it's "

Return an RDD that yields (path, line) for each line in the file. *path* must be a single path, not a comma-separated list of paths

Sort key to sort the given dictionaries containing IDs roughly by time (earliest first). We consider higher attempt_nums "later" than higher

Convert a value (usually for a configuration property) into its Java string representation, falling back to the Python representation if None

Take in data as a sequence of bytes, and yield it, one line at a time. Only breaks lines on ``\\n`` (not ``\\r``), and does not add a trailin

If *spark_error* contains a stack trace followed by a blank line, trim off the blank line and everything that follows.

Return where *path* will be uploaded, or, if it starts with ``'file:///'``, a local path.

Return a dictionary with the same keys as *d*, but where the non-string, non-bytes values have been JSON-encoded. Used to encode cluster prop

Helper for _wait_for_step_to_complete(). Wait for step with the given ID to complete, and fetch counters. If it fails, attempt to diag

Helper for _wait_for_step_to_complete(). Wait for step with the given ID to complete, and fetch counters. If it fails, attempt to diag

When we tell Hadoop/Spark to put files in the working directory, must they have the same names as the files in the working dir? This

Write text to the given file. By default, this writes binary data, but can be redefined to use local line endings.

Get relevant cluster information from EMR. :param float max_days_ago: If set, don't fetch clusters created longer

Get a list of IDs of pooled clusters that this runner can join, sorted so that the ones with the greatest CPU capacity come first. yi

Use curl to perform an HTTP GET on the given path on the YARN Resource Manager. Either return decoded JSON from the call, or raise an

Like :py:attr:`ARCHIVES`, except that it can return a dynamically generated list of archives to upload and unpack. Overriding this met

Decompress gzipped data on the fly. :param fileobj: object supporting ``read()`` :param bufsize: number of bytes to read from *fileobj* at a

Mock SSH behavior for running cat <path> over SSH

Clean up running jobs, temp files, and logs, subject to the *cleanup* option passed to the constructor. If you create your runner in

This method assumes ``prepare_runner_for_ssh()`` was called. That method isn't a "proper" setup method because it requires different a

Define arguments for this script. Called from :py:meth:`__init__()`. Re-define to define custom command-line arguments or pass throug

Break text down into ngrams, and return a dictionary mapping (n, ngram) to number of times that ngram occurs. n: ngram size ("foo" is a 1-gra

Dump a wrapper script to the given file object that runs this python script.

Like :py:attr:`DIRS`, except that it can return a dynamically generated list of directories to upload. Overriding this method disables

Like :py:attr:`FILES`, except that it can return a dynamically generated list of files to upload. Overriding this method disables :py:

Get a path to a directory containing part files to use as mock output.

Look for a usable instance profile for EMR, and if there is none, create one.

Look for a usable service role for EMR, and if there is none, create one. Either way, return that role's name.

Decompress gzipped data on the fly. :param fileobj: object supporting ``read()`` :param bufsize: number of bytes to read from *fileobj* at a

``-D`` args to pass to hadoop streaming. This should be a map from property name to value. By default, returns :py:attr:`JOBCONF`. ..

Optional list of paths of jar files to run our job with using Hadoop's ``-libjars`` option. Normally setting :py:attr:`LIBJARS` is suf

stripped-down simulation of list_instances() to support SSH tunneling; only includes state.status and the privateipaddress field.

Load command-line options into ``self.options``. Called from :py:meth:`__init__()` after :py:meth:`configure_args`. :type args: list

Mock SSH behavior for running the ``find`` command over SSH

Run the create_cluster tool with arguments from ``sys.argv`` and printing to ``sys.stdout``.

Implements hadoop <args>

Create a :py:mod:`boto3` EC2 client. :return: a :py:class:`botocore.client.EC2` wrapped in a :py:class:`mrjob.retry.RetryWrap

Does not actually create a directory on GCS (because GCS doesn't have directories), but creates the underlying bucket if it does not e

Make a directory. This doesn't actually create directories on S3 (because there is no such thing), but it will create the corresponding