Check sh_bin

Fix and check various EMR-specific options

Treat streaming steps as Spark steps.

Just run tasks in the same process.

A pickleable function that invokes a task in a subprocess.

Used to disable S3Filesystem when boto3 is installed but credentials aren't set up.

Used to identify task errors.

Is the exception from a boto3 client retriable?

Just tell what file we were reading from (since they'll see the stacktrace from the actual exception)

Utility for logging counters (if any) for a step.

Log log4j record parsed from hadoop stderr.

sort key which treats empty timestamp as most recent

Obfuscate AWS credentials.

Combine *cmdenv* with :py:func:`~mrjob.conf.combine_local_envs`

Call no-args function *func*, returning *None* and ensuring that any exception raised is pickleable.

Return a dict with "hidden" tags to add to the given cluster.

Patch in :py:attr:`_job_script_zip_path`, if running streaming steps.

Helper for :py:meth:`SimMRJobRunner._run_mapper_and_combiner_func`.

Use multiprocessing to run in parallel.

Set up a fake working directory and environment, and call the Spark method.

Set up filehandles and call *invoke_task()*. Helper for :py:meth:`SimMRJobRunner._run_task_func`.

Try sorting with the :command:`sort` binary before falling back to in-memory sort.

Sort lines the given *input_paths* into *output_path*, using *sort_bin*. If there is a problem, fall back to in-memory sort. This is a helper

Does the spark driver have a working directory different from the one *spark-submit* was run in? (Only true in cluster mode.)

Generate spark harness args for streaming steps (and handle other spark step types the usual way).

For streaming steps, return the path of the harness script (and handle other spark step types the usual way).

Wait 20 seconds because gcloud has to update project metadata (unless we were going to check the cluster sooner anyway).

Sort by timestamp and step

Treat streaming steps as Spark steps that use Python.

Yield lists of directories to look for the history log in.

Yield lists of directories to look for the task logs in.

Write text to the given file, using local line endings.

Add a mocked worker to the cluster. Caller is responsible for setting runner._opts['num_ec2_instances'] to the correct number.