Convert a pipeline or list which contains sparknlp/sparknlp_jsl annotators into NLU pipeline, while maintaining original configuration. T

Resolve a Spark NLU reference to a NLP reference. :param nlu_ref: which nlu model_anno_obj's nlp refrence to return. :param lang: what

For a given anno-class returns NLU-Component which wraps the corresponding pipe class but has no model_anno_obj yet loaded onto it. :para

Set Secret environ variables for Spark Context

Validate for iterable data input if all elements point to file or jsl_folder

Validate for iterable data input if all elements point to file or jsl_folder

This method configures sentenceEmbeddings and Classifier components to output at a specific level. Generally this substitutes all `se

Find first embed component_to_resolve in component_list

Find first occurrence of component in pipe by ID and returns index

Check if component_list is trainable

Extract storage ref from a NLU component_to_resolve which embellished a Spark NLP Annotator

Annotates a Pandas Dataframe/Pandas Series/Numpy Array/Spark DataFrame/Python List strings /Python String abd returns List of Floats or Spark

Casting question pandas to spark and add index col

Casting question pandas to spark and add index col

Casting str to spark and add index col. This is a bit inefficient. Casting follow # inefficient, str->pd->spark->pd , we can could first pd

Casting str to spark and add index col. This is a bit inefficient. Casting follow # inefficient, str->pd->spark->pd , we can could first pd

Convert supported datatypes to SparkDF and extract extra data for prediction later on.

Write metadata fields to pipeline, for now only whether it contains OCR components or not. To be extended in the future

apply extract_master on all fields with corrosponding configs after converting Pyspark Rows to List[Dict] and merge them to a final DF (1 to 1 ma

Set Secret environ variables for Spark Context

Configure component_list components to output level document. Substitute every occurence of <sentence> to <document> for every component_to_r

for every component_to_resolve, set its inputs and outputs to the ones configured on the NLU component_to_resolve.

Extract an iterable object containing paths from input data

Find col that contains embed

Get's a list of all columns that have been derived in the pythonify procedure from the component_to_resolve os_components in dataframe df for

The tail of a NLU ref after splitting on '.' gives a unique identifier for NON-Aliased components If result is '' , model_anno_obj UID will b

Resolves a URL for an NLU reference. :param nlu_ref: Which nlu refrence's url to return. :return: url to modelhub

Get all paths which point to correct file types from iterable paths which can contain file and jsl_folder paths 1. paths point to a file which

Check for NLUPipeline if it contains sentence detector

Check for NLUPipieline if it contains any table extracting OCR component

Check if a NLU Component consumes embeddings