Infer embedding dimension from embedding file Args: embedding_file (str): local embedding file path Returns:

Load from vocab file Args: vocab_file (str): vocab file to be loaded Returns: word2id (Dict[str, int]): a di

TODO

Apply a given function to reformat a `Dataset`.

The given `indices` of size `(batch_size, d_1, ..., d_n)` indexes into the sequence dimension (dimension 2) of the target, which has size `(b

label sequence to span format

Build decoder from config dict Args: cfg (:obj:`ConfigDict`): config dict for decoder object default_args (dict): default initial

Build embedder from config dict Args: cfg (:obj:`ConfigDict`): config dict for embedder object default_args (dict): default initi

Build encoder from config dict Args: cfg (:obj:`ConfigDict`): config dict for encoder object default_args (dict): default initial

Build lr scheduler, `constant` by default.

Build optimizer from config

Entrypoint of build the trainer from `config` by modelscope. In this method, we will build the `DatasetManager` first, then use the count

Entrypoint of build the trainer from `config` by modelscope. In this method, we will build the `DatasetManager` first, then use the count

Compute f1 from predictions and ground truth Args: preds (List[str]): prediction list labels (List[str]): ground truth list

convert label sequence to span

:param span_scores: (b, t, t, c) :param mask: (b, t) :return:

encode `text` to ids

encode tokens when using retrieval-augmented multi-view model.

Convert tokens to ids, one by one via vocab, no word pieces.

Convert tokens to ids by word piece tokenizer.

TODO

fix label sequence errors

This is a subroutine for [`batched_index_select`](./util.md#batched_index_select). The given `indices` of size `(batch_size, d_1, ..., d_n)`

Build embedder instance from config

produce label embeddings

TODO

Get member names set.

get model cache dir

Returns a transformer model and a flag of whether comes from huggingface. # Parameters model_name_or_path : `str` The name of t

load pretrained model

Utils function to obtain glove embedding

Takes a list of model parameters with associated names (typically coming from something like `model.named_parameters()`), along with a groupi

Utils function to obtain tencent zh embedding

Multi-label cross entropy loss. https://kexue.fm/archives/7359

pad other fields.

pad token related fields (hf.transformers style)

Run something after __init__ in subclass All derived model instances will try to load checkpoint from model_dir after __init__. Usefu

Prepare global logging.

read and parse yaml file :param str file_path: path to file :param dict cfg: configuration variables (environ and .env) :param bool stric

Initialize the transition parameters. The parameters will be initialized randomly from a uniform distribution between -0.1 and 0.1.

Initialize the transition parameters. The parameters will be initialized randomly from a uniform distribution between -0.1 and 0.1.

Command line main interface

convert all possible fields to tensor.

Train a model from config file. You can mannualy call this function in a python script for debugging.

Override this func to build task dataset from only `datasets.Dataset`.

Train a model from config file or dict. You can manually call this function in a python script for debugging.

TODO

pad list of instances to batch

prepare inputs for Pretraining model

prepare inputs for two-stage-ner model

prepare inputs for Sequence Labeling models.

prepare inputs for span-based model.

Encode one instance, it could be a text str, a list of tokens for a dict. Returns: Dict[str, Any]: `{'tokens': tokenized

prepare inputs for Entity Typing model

prepare inputs for Relation Extraction model.