Functions — github.com/awslabs/mlm-scoring

hub / github.com/awslabs/mlm-scoring / functions

Functions116 in github.com/awslabs/mlm-scoring

⨍ Functions 116◇ Types & classes 26

↓ 12 callersFunctionget_pretrained

(ctxs: List[mx.Context], name: str = 'bert-base-en-uncased', params_file: Optional[Path] = None, cased: bool =

src/mlm/models/__init__.py:87

↓ 10 callersMethodfrom_file

(cls, fp: TextIO, **kwargs)

src/mlm/loaders.py:73

↓ 7 callersMethodfrom_dict

Loads reference texts from the format of Shin et al. (JSON) Args: fp (TextIO): JSON file object

src/mlm/loaders.py:84

↓ 5 callersMethod_apply_tokenizer_opts

(self, sent: str)

src/mlm/scorers.py:52

↓ 5 callersFunctionsetup_ctxs

(gpu_str: str)

src/mlm/cmds.py:49

↓ 4 callersMethod__init__

(self, *args, **kwargs)

src/mlm/scorers.py:172

↓ 4 callersMethod_check_support

(model)

src/mlm/scorers.py:181

↓ 4 callersFunction_shared_args

(parser: argparse.ArgumentParser)

src/mlm/cmds.py:25

↓ 3 callersMethod_corpus_to_data

(self, corpus, split_size, ratio, num_workers: int, shuffle: bool=False)

src/mlm/scorers.py:65

↓ 3 callersMethod_split_batch

(self, batch)

src/mlm/scorers.py:97

↓ 3 callersMethod_true_tok_lens

(self, dataset)

src/mlm/scorers.py:82

↓ 3 callersMethodcorpus_to_dataset

(self, corpus: Corpus)

src/mlm/scorers.py:185

↓ 3 callersMethodfrom_corpus_and_scores

(cls, corpus: Corpus, scores: List[float])

src/mlm/loaders.py:363

↓ 3 callersMethodfrom_text

(cls, fp: Iterable[str], max_utts=None)

src/mlm/loaders.py:108

↓ 3 callersFunctionget_model

Returns a pre-defined model by name. In addition to the models in GluonNLP model API, this API supports getting GPT-2 models. Parameters

src/mlm/models/__init__.py:20

↓ 3 callersMethodscore

(self, corpus: Corpus, temp: float = 1.0, split_size: int = 2000, ratio: float = 0, num_workers: int = 10, per

src/mlm/scorers.py:419

↓ 3 callersMethodto_corpus

(self)

src/mlm/loaders.py:332

↓ 3 callersMethodto_file

(self, fp: TextIO, scores_only: bool = False)

src/mlm/loaders.py:387

↓ 3 callersMethodto_json

(self, fp: TextIO)

src/mlm/loaders.py:342

↓ 2 callersMethod__init__

(self, units, vocab_size, max_length, num_layers, num_heads, dropout=0.0, prefix=None, params

src/mlm/models/gpt2.py:213

↓ 2 callersFunction_get_gpt2_model

Any predefined GPT-2 model. Parameters ---------- model_name : str or None, default None Options include 'gpt2_117m' and 'gpt2_34

src/mlm/models/gpt2.py:381

↓ 2 callersFunction_get_json_fp

(tmp_path)

tests/test_loaders.py:26

↓ 2 callersMethod_get_num_words_in_utt

(self, utt: str)

src/mlm/loaders.py:121

↓ 2 callersFunction_get_scorer_and_corpus_eos

()

tests/test_scorers.py:13

↓ 2 callersMethodfrom_nmt

Loads hypotheses from Toan's NMT beam output format Args: fp (TextIO): .nobpe filename max_utts (None, option

src/mlm/loaders.py:279

↓ 2 callersMethodscore_sentences

(self, sentences: List[str], **kwargs)

src/mlm/scorers.py:165

↓ 2 callersMethodto_predictions

(self)

src/mlm/loaders.py:396

↓ 1 callersMethod__init__

(self, bert, dropout=0.0, prefix=None, params=None)

src/mlm/models/bert.py:55

↓ 1 callersMethod_batch_ops

(self, batch, batch_sent_idxs_per_ctx, batch_scores_per_ctx, temp, per_token=False)

src/mlm/scorers.py:203

↓ 1 callersMethod_batch_ops

(self, batch, batch_sent_idxs_per_ctx, batch_scores_per_ctx, temp)

src/mlm/scorers.py:908

↓ 1 callersMethod_bin_ops

(self, batch, bin_counts_per_ctx, bin_sums_per_ctx, temp)

src/mlm/scorers.py:249

↓ 1 callersMethod_check_support

(model)

src/mlm/scorers.py:61

↓ 1 callersMethod_edit_distance

(a: str, b: str)

src/mlm/loaders.py:140

↓ 1 callersMethod_generate_ln

(self, alpha=0.6, tokenizer=None, ln_type='gnmt')

src/mlm/loaders.py:27

↓ 1 callersFunction_get_text_fp

(tmp_path)

tests/test_loaders.py:34

↓ 1 callersMethod_ids_to_masked

(self, token_ids: np.ndarray)

src/mlm/scorers.py:334

↓ 1 callersMethod_ids_to_masked

(self, token_ids: np.ndarray)

src/mlm/scorers.py:576

↓ 1 callersFunction_mbleu

(file_ref: TextIO, preds_hyps: Predictions)

src/mlm/cmds.py:404

↓ 1 callersFunction_pad_arrs_to_max_length

Inner Implementation of the Pad batchify Parameters ---------- arrs : list pad_axis : int pad_val : number use_shared_mem : b

src/mlm/batchify.py:34

↓ 1 callersFunction_stack_arrs

(arrs, use_shared_mem, dtype)

src/mlm/batchify.py:86

↓ 1 callersFunction_stdout_to_stderr

()

src/mlm/cmds.py:37

↓ 1 callersFunction_wer

(file_ref: TextIO, preds_hyps: Predictions)

src/mlm/cmds.py:392

↓ 1 callersMethodbin

(self, corpus: Corpus, temp: float = 1.0, split_size: int = 2000, ratio: float = 1, num_workers: int = 10)

src/mlm/scorers.py:275

↓ 1 callersMethodcorpus_to_dataset

(self, corpus: Corpus)

src/mlm/scorers.py:376

↓ 1 callersMethodcorpus_to_dataset

(self, corpus: Corpus)

src/mlm/scorers.py:607

↓ 1 callersMethoddetok_copy

(self)

src/mlm/loaders.py:202

↓ 1 callersMethodfrom_files

Creates a ScoredCorpus from separate text and score files Args: corpus_file (Path): A file with a sentence per line s

src/mlm/loaders.py:371

↓ 1 callersMethodget_num_words

(self)

src/mlm/loaders.py:125

↓ 1 callersMethodmbleu_score

(self, preds: Any)

src/mlm/loaders.py:178

↓ 1 callersMethodrescore

This implements rescoring as: s_final = (1-scale)*s_orig + scale*s_new Args: new_scores (TYPE): List of list of n

src/mlm/loaders.py:39

↓ 1 callersMethodscore

(self, corpus: Corpus, temp: float = 1.0, split_size: int = 2000, ratio: float = 1, num_workers: int = 10, per

src/mlm/scorers.py:101

↓ 1 callersMethodtune

(self, scored_corpus: ScoredCorpus, temp: float = 1.0, split_size: int = 2000, ratio: float = 0, num_workers:

src/mlm/scorers.py:947

↓ 1 callersMethodwer_score

(self, preds: Any)

src/mlm/loaders.py:167

Method__call__

Batchify the input data Parameters ---------- data : list The input data samples Returns -------

src/mlm/batchify.py:154

Method__call__

Batchify the input data. The input can be list of numpy.ndarray, list of numbers or list of mxnet.nd.NDArray. Inputting mxnet.nd.NDAr

src/mlm/batchify.py:246

Method__call__

Batchify the input data. Parameters ---------- data : list The samples to batchfy. Each sample should contain N a

src/mlm/batchify.py:338

Method__call__

Parameters ---------- data The list of samples Returns ------- ret The input

src/mlm/batchify.py:375

Method__call__

Parameters ---------- data The samples to batchify. Each sample should be a dictionary Returns

src/mlm/batchify.py:435

Method__call__

Batchify the input data. Parameters ---------- data The samples to batchfy. Each sample should be a namedtuple.

src/mlm/batchify.py:534

Method__init__

(self, sents: List[str], scores: List[float], vocab: Optional[nlp.Vocab] = None, tokenizer: Any = None)

src/mlm/loaders.py:20

Method__init__

(self, dtype=None)

src/mlm/batchify.py:151

Method__init__

(self, axis=0, pad_val=None, ret_length=False, dtype=None, round_to=None)

src/mlm/batchify.py:228

Method__init__

(self, fn, *args)

src/mlm/batchify.py:326

Method__init__

(self, fn_dict: t_Dict[AnyStr, t_Callable])

src/mlm/batchify.py:425

Method__init__

(self, container: t_NamedTuple, fn_info: t_Union[t_List[t_Callable],

src/mlm/batchify.py:509

Method__init__

(self, model: Block, vocab: nlp.Vocab, tokenizer, ctxs: List[mx.Context], eos: Optional[bool] = None, capitali

src/mlm/scorers.py:29

Method__init__

(self, *args, **kwargs)

src/mlm/scorers.py:322

Method__init__

(self, *args, **kwargs)

src/mlm/scorers.py:546

Method__init__

(self, *args, **kwargs)

src/mlm/scorers.py:875

Method__init__

(self, *args, **kwargs)

src/mlm/scorers.py:988

Method__init__

(self, units, num_heads, dropout=0.0, weight_initializer=mx.init.Normal(0.02), bias_initializ

src/mlm/models/gpt2.py:73

Method__init__

(self, units, hidden_size, weight_initializer=mx.init.Normal(0.02), bias_initializer='zeros',

src/mlm/models/gpt2.py:171

Method__init__

(self, bert, num_classes=2, dropout=0.0, p

src/mlm/models/bert.py:107

Method_batch_ops

(self, batch, batch_sent_idxs_per_ctx, batch_scores_per_ctx, temp)

src/mlm/scorers.py:1019

Method_check_support

(model)

src/mlm/scorers.py:330

Method_check_support

(model)

src/mlm/scorers.py:572

Method_check_support

(model)

src/mlm/scorers.py:997

Method_word_len

(sent: str)

src/mlm/loaders.py:163

Methodbin

(self, corpus: Corpus, temp: float = 1.0, split_size: int = 2000, ratio: float = 0, num_workers: int = 10)

src/mlm/scorers.py:787

(args: argparse.Namespace)

src/mlm/cmds.py:416

Functioncmd_rescore

rescore command You have two files with the following schema: { "<UTT_ID>": { "ref": { "score": 0.111

Methodcorpus_to_dataset

(self, corpus: ScoredCorpus)

src/mlm/scorers.py:890

Methodcorpus_to_dataset

(self, corpus: Corpus)

src/mlm/scorers.py:1001

Methodforward

(self, data, states=None)

src/mlm/models/gpt2.py:93

Methodforward

Parameters ---------- data : NDArray Shape (batch_size, seq_len) states : list of NDArray or None

src/mlm/models/gpt2.py:242

Methodforward

Generate the unnormalized score for the given the input sequences. Parameters ---------- inputs : NDArray, shape (batch_size,

src/mlm/models/bert.py:64

Methodforward

Generate the unnormalized score for the given the input sequences. Parameters ---------- inputs : NDArray, shape (batch_size,

src/mlm/models/bert.py:121

Methodforward

r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked l

src/mlm/models/bert.py:148

Methodforward

r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked l

src/mlm/models/bert.py:222

Methodforward

r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Labels for computing the masked l

src/mlm/models/bert.py:303

Methodfrom_dict

Loads hypotheses from the format of Shin et al. (JSON) Args: fp (str): JSON file name max_utts (None, optiona

src/mlm/loaders.py:233

Methodfrom_file

(cls, fp: TextIO, **kwargs)

src/mlm/loaders.py:221

Functiongpt2_117m

Generic GPT-2 model. The number of layers (L) is 12, number of units (H) is 768, and the number of self-attention heads (A) is 12. Param

src/mlm/models/gpt2.py:313

Functiongpt2_345m

Generic GPT-2 model. The number of layers (L) is 24, number of units (H) is 1024, and the number of self-attention heads (A) is 24. Para

src/mlm/models/gpt2.py:347

Methodhybrid_forward

(self, F, data)

src/mlm/models/gpt2.py:186

Functionmain

Defines arguments for all subcommands

src/mlm/cmds.py:72

Methodprint_record

(self, record)

src/mlm/scorers.py:366

Methodscore

(self, corpus: Corpus, temp: float = 1.0, split_size: int = 2000, ratio: float = 0, per_token: bool = False)

src/mlm/scorers.py:633

next →1–100 of 116, ranked by callers