Returns a list containing all target egg_files Returns: list: target egg_files

test

Find all globals names read or written to by codeblock co

inner function

Get fs.defaultFS from path like "hdfs://abcde:22222/a/b/c"

Turn the function into a tuple of data necessary to recreate it: code, globals, defaults, closure, dict

Wraps console command 'hadoop fs -mkdir -p <path>' Args: path (str): path to be created

用户可以重写该方法。 返回一个commiter, 默认表示不需要commit阶段 commiter应该是一个无参函数。

get_fields

Get the corresponding node of input Returns: LogicalPlan.Node: input node

used to get mock client

get_msg_type

User can override this method to set the serde

内部函数

Send the rpc command to the other end to get the logical plan status Args: Raises: error.BigflowRPCException: if any error happe

获取当前的运行状态 Returns: str: APP_SUBMIT/APP_ALLOCATE/APP_RUN/APP_KILL/APP_FAIL/APP_UNKNOWN AM

利用给定的key_extractor和value_extractor对输入PCollection分组，返回一个表示分组结果的PTable Args: pcollection (PCollection): 输入PCollection key_extract

init_log - initialize log module Args: level (str): msg above the level will be displayed DEBUG < INFO < WARNIN

Plug in system. Register additional pickling functions if modules already loaded

Block email LazyImporters from being saved

Handle bugs with pickling scikits timeseries

Check if a given path is HDFS uri Args: path (str): input path Returns: bool: True if input is a HDFS path, False otherwi

Check if a given toft style path is HDFS path Args: path (str): input path Returns: bool: True if input is a HDFS path, F

Send the rpc command to the other end to kill the logical plan Args: Raises: error.BigflowRPCException: if any error happened

Send the rpc command to the other end to launch the logical plan Args: Raises: error.BigflowRPCException: if any error happened

inner

Implementation of transforms.max()

inner

inner

inner

flush all datasets with preceding key, and push datasets with same key return true if enter new group

return name

inner

group by fn, for internal use

group by window, for internal use

Case: test combinations of operators

inner function

用户可以重写该方法。 返回一个partition fn。 partition_fn原型应为：(data, total_partition) => partition 返回None则表示不太关心如何partition。

用户可以重写该方法。 返回一个int型的数，表示总共要把数据partition成多少份。