Get the output feature dim.

Instantiate a model from config dict or path. Args: cfg_dict_or_path (Optional[Union[str, Dict, Config]]): config dict or path

Instantiate a model from local directory or remote model repo. Note that when loading from remote, the model revision can be specified.

dump predictions

Return (sub-)token mask that removed special tokens.

add predictions to cache

move to device

# Parameters input_ids: `torch.LongTensor` Shape: `[batch_size, num_wordpieces]`. attention_mask: `torch.BoolTen

Convert tokens to ids, add some mask.

Try to load model checkpoint from model_dir/pytorch_model.bin

Args: emissions (`~torch.Tensor`): Emission score tensor of size ``(seq_length, batch_size, num_tags)``

build data collator from config.

Build preprocessor from config

build trainer from config

Get the input feature dim.

Check whether a nested dict has a key Args: _dict (Dict): a nested dict like object *keys (str): flattened key list Returns:

Check if a directory is empty

Generate `label_to_id` mapping. You can override this method to customize the mapping. NOTE: The `self.labels` could be modif

sum with mask

# Parameters input_ids: `torch.LongTensor` Shape: [batch_size, num_wordpieces]. offsets: `torch.LongTensor`

example: token label the O <E> /misc/misc/part_of Circle B-A Undone I-A </E> O

Resolve datasets. From (remote) data_dir or data_files Args: dl_manager: Returns: data_files

Mask X according to the mask.

Update counters.

Add testing arguments parser

label sequence to span format

build tokenizer from `transformers`.

Compute posterior probability distribution from emission logits Args: emissions (`~torch.Tensor`): Emission score tensor of size

Create a string indicating current time Create a string indicating current time in microsecond precision, for example, 221109144626.861616

Dump dict to log file

Returns the device of the tensor.

Get file by keyword, such as: train/test/dev.

see `get_transformer`.

see `get_transformer`.

Returns a range vector with the desired size, starting at 0. The CUDA implementation is meant to avoid copy data from CPU to GPU.

Prepare logging for training, log to file in `work_dir`.

Calculate loss for two-stage model

targets : (batch_size, num_classes, seq_len, seq_len) entity_score : (batch_size, num_classes, seq_len, seq_len)

span_labels : (batch_size, seq_len, seq_len) span_scores : (batch_size, seq_len, seq_len, num_classes)

Parameters: emissions: (seq_length, batch_size, num_tags) tags: (seq_length, batch_size) mask: (seq_lengt