Merge token feature generators into embed component_list. i.e. Pos/Dep_depdency/Untyped_dep if not already present in component_list

Check if a storage ref is defined on the Spark NLP Annotator model_anno_obj

Check if a storage ref is defined on the Spark NLP Annotator embellished by the NLU Component

This method implements the main namespace for all component_to_resolve names. It parses the input request and passes the data to a resolver m

Casting numpy array to spark and add index col. This is a bit inefficient. Casting follow np->pd->spark->pd. We could cut out the first pd step

Casting pandas series to spark and add index col. # for df['text'] colum/series passing casting follows pseries->pdf->spark->pd

Print all languages which are available in NLU Spark NLP pointer

Remove required storage ref features if conversion candidate has it, so that storage ref provider will not be downloaded twice

Rename cols with duplicate names

Write metadata fields to pipeline, for now only whether it contains OCR components or not. To be extended in the future :return:

For a given NLU component which is input dependent , resolve its output level by checking if it's input stem from document or senten

Returns a nlp_ref, nlu_ref and whether it is a licensed model_anno_obj or not and an updated languiage, if multi lingual

Set output/input cols on Nlu Components loaded from a pipeline

For every embedding converter, we set storage ref attr on it, based on what the storage ref from it's provider is

Convert data to LihgtPipeline Compatible Format, which is np.array[str], list[str] and str but we need list anyways later. So we cre

Casting str list to spark and add index col. This is a bit inefficient. Casting follow # # inefficient, list->pd->spark->pd , we can could first pd

Casting str to spark and add index col. This is a bit inefficient. Casting follow # inefficient, str->pd->spark->pd , we can could first pd

Some truly irrelevant cols might be dropped, regardless of anno Extractor config Some truly irrelevant cols might be dropped, regardl

Convert data to LihgtPipeline Compatible Format, which is np.array[str], list[str] and str but we need list anyways later. So we cre

1. Unpack SDF to PDF with Spark NLP Annotator Dictionaries 2. Get the extractor configs for the corresponding Annotator classes

Some models have bad storage refs. The list of these bad models is defined by nlu.spellbook.Spellbook.bad_storage_refs. The correct s

Storage ref of converters is initially empty string, i.e. '' . This method checks if any convertable embeddings are provided, if yes it will

Validate for input data that it contains a path pointing to file or jsl_folder

Validate for input data that it contains a path pointing to file or folder of audio fila readable with librosa

Load a licensed model_anno_obj from HDD

Vizualize licensed component_to_resolve

Vizualize open source component_to_resolve

Viz relation result. Set label colors by specifying hex codes, i.e. viz_colors={'TREATMENT':'#008080', 'problem':'#800080'}

Viz relation result. Set label colors by specifying hex codes, i.e. viz_colors={'TREATMENT':'#800080', 'PROBLEM':'#77b5fe'}

Viz dep result. Set label colors by specifying hex codes, i.e. viz_colors={'TREATMENT':'#800080', 'PROBLEM':'#77b5fe'}

Display Viz in streamlit

Visualize either individual building blocks for streamlit or a full UI to experiment and explore models with

Wrap function with ST cache method if streamlit is importable

Returns a new dataframe, where columns in cols_to_explode should have all array elements. :param df: Dataframe to explode columns on. Each co