Functions — github.com/huggingface/transformers

hub / github.com/huggingface/transformers / functions

Functions4,119 in github.com/huggingface/transformers

⨍ Functions 4,119◇ Types & classes 939↳ Endpoints 4

↓ 2 callersFunctionpickle_save

pickle.dump(obj, path)

examples/seq2seq/utils.py:188

↓ 2 callersMethodpostprocess_next_token_scores

( self, scores, input_ids, no_repeat_ngram_size, bad_words_ids,

src/transformers/modeling_utils.py:828

↓ 2 callersMethodprepare_config_and_inputs_for_decoder

(self)

tests/test_modeling_mobilebert.py:129

↓ 2 callersMethodprepare_inputs_for_generation

(self, input_ids, **kwargs)

src/transformers/modeling_utils.py:804

↓ 2 callersMethodprepare_inputs_for_generation

(self, inputs, **kwargs)

src/transformers/modeling_tf_utils.py:541

↓ 2 callersMethodpresign_and_upload

Get a presigned url, then upload file to S3. Outputs: url: Read-only url for the stored file on S3.

src/transformers/hf_api.py:150

↓ 2 callersFunctionprint_2d_tensor

Print a 2D tensor

examples/bertology/run_bertology.py:56

↓ 2 callersMethodprune_heads

(self, heads)

src/transformers/modeling_xlm.py:111

↓ 2 callersMethodprune_heads

(self, heads)

examples/movement-pruning/emmental/modeling_bert_masked.py:203

↓ 2 callersFunctionremove_ignore_keys_

(state_dict)

src/transformers/convert_bart_original_pytorch_checkpoint_to_pytorch.py:56

↓ 2 callersMethodreset_length

(self, tgt_len, ext_len, mem_len)

src/transformers/modeling_transfo_xl.py:708

↓ 2 callersMethodround_batch

For float16 only. Sub-sample sentences in a batch, and add padding, so that each dimension is a multiple of 8. Input:

examples/distillation/distiller.py:283

↓ 2 callersFunctionrun_generate

()

examples/seq2seq/run_eval.py:60

↓ 2 callersMethodsanitize_special_tokens

Make sure that all the special tokens attributes of the tokenizer (tokenizer.mask_token, tokenizer.cls_token, ...) are in the vocabulary.

src/transformers/tokenization_utils_base.py:610

↓ 2 callersMethodsave_binary

Save the provided data object as a pickle-formatted binary data on the disk. :param data: data to store :return: (str) Path w

src/transformers/pipelines.py:197

↓ 2 callersFunctionsave_json

(data, path: str)

src/transformers/tokenization_marian.py:233

↓ 2 callersFunctionsave_json

(content, path)

examples/seq2seq/utils.py:204

↓ 2 callersMethodsave_token

Save token, creating folder as needed.

src/transformers/hf_api.py:234

↓ 2 callersMethodset_embed_tokens

(self, embed_tokens)

src/transformers/modeling_tf_t5.py:544

↓ 2 callersMethodset_input_embeddings

(self, new_embeddings)

src/transformers/modeling_t5.py:652

↓ 2 callersFunctionset_seed

(args)

templates/adding_a_new_example_script/run_xxx.py:67

↓ 2 callersFunctionset_seed

(args)

examples/contrib/run_swag.py:216

↓ 2 callersFunctionset_seed

(args)

examples/contrib/mm-imdb/run_mmimdb.py:56

↓ 2 callersFunctionset_seed

(args)

examples/distillation/run_squad_w_distillation.py:80

↓ 2 callersFunctionset_seed

(args)

examples/movement-pruning/masked_run_glue.py:62

↓ 2 callersFunctionset_seed

(args)

examples/movement-pruning/masked_run_squad.py:66

↓ 2 callersFunctionset_seed

(args)

examples/text-classification/run_xnli.py:55

↓ 2 callersFunctionset_seed

(args)

examples/bert-loses-patience/run_glue_with_pabee.py:63

↓ 2 callersFunctionset_seed

(args)

examples/question-answering/run_squad.py:62

↓ 2 callersMethodset_seed

(self)

tests/test_modeling_transfo_xl.py:82

↓ 2 callersMethodset_seed

(self)

tests/test_modeling_tf_transfo_xl.py:86

↓ 2 callersFunctionshift_tokens_right

Shift input ids one token to the right, and wrap the last non pad token (usually <eos>).

src/transformers/modeling_bart.py:177

↓ 2 callersFunctionstart_memory_tracing

Setup line-by-line tracing to record rss mem (RAM) at each line of a module or sub-module. See `../../examples/benchmarks.py for a usage exam

src/transformers/benchmark/benchmark_utils.py:306

↓ 2 callersFunctionstop_memory_tracing

Stop memory tracing cleanly and return a summary of the memory trace if a trace is given. Args: - `memory_trace` (optional outpu

src/transformers/benchmark/benchmark_utils.py:453

↓ 2 callersMethodstream_iterator

(self, sent_stream)

src/transformers/tokenization_transfo_xl.py:576

↓ 2 callersMethodto_dict

Serializes this instance to a Python dictionary. Returns: :obj:`Dict[str, any]`: Dictionary of all the attributes that m

src/transformers/configuration_utils.py:357

↓ 2 callersMethodto_json_string

Serializes this instance to a JSON string. Args: use_diff (:obj:`bool`): If set to True, only the differ

src/transformers/configuration_utils.py:369

↓ 2 callersMethodtokenize

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi

src/transformers/tokenization_bert.py:498

↓ 2 callersMethodtranslate_batch

Translate a batch of sentences. Mostly a wrapper around :obj:`Beam`. Args: batch (:obj:`Batch`): a batch from a

examples/seq2seq/bertabs/modeling_bertabs.py:781

↓ 2 callersMethodtranspose_for_scores

(self, x, batch_size)

src/transformers/modeling_tf_albert.py:188

↓ 2 callersMethodtranspose_for_scores

(self, x)

src/transformers/modeling_bert.py:207

↓ 2 callersFunctionu

(e,t,o)

docs/source/_static/js/custom.js:246

↓ 2 callersFunctionx

(e,t,o)

docs/source/_static/js/custom.js:246

↓ 1 callersFunctionC

(e)

docs/source/_static/js/custom.js:246

↓ 1 callersMethod__getstate__

(self)

src/transformers/tokenization_utils_base.py:189

↓ 1 callersMethod__init__

( self, vocab_file, merges_file, errors="replace", bos_token="<s>",

src/transformers/tokenization_roberta.py:125

↓ 1 callersMethod__init__

(self, vocab_file, merges_file, unk_token="<unk>", **kwargs)

src/transformers/tokenization_openai.py:100

↓ 1 callersMethod__init__

( self, initial_learning_rate, decay_schedule_fn, warmup_steps, power=1.0, name=None, )

src/transformers/optimization_tf.py:26

↓ 1 callersMethod__init__

( self, vocab_file, merges_file, errors="replace", unk_token="<|endoft

src/transformers/tokenization_gpt2.py:141

↓ 1 callersMethod__init__

Constructs a MecabTokenizer. Args: **do_lower_case**: (`optional`) boolean (default True) Whether to lower case t

src/transformers/tokenization_bert_japanese.py:170

↓ 1 callersMethod__init__

( self, vocab_file, do_lower_case=True, do_basic_tokenize=True, never_

src/transformers/tokenization_bert.py:164

↓ 1 callersMethod__init__

( self, data: Optional[Dict[str, Any]] = None, encoding: Optional[Union[EncodingFast,

src/transformers/tokenization_utils_base.py:145

↓ 1 callersMethod__init__

(self, args: BenchmarkArguments = None, configs: PretrainedConfig = None)

src/transformers/benchmark/benchmark_utils.py:566

↓ 1 callersMethod__init__

Reads source and target sequences from txt files.

examples/text-generation/pplm/run_pplm_discrim_train.py:85

↓ 1 callersMethod__init__

(self, data, batch_size)

examples/seq2seq/utils.py:152

↓ 1 callersMethod__init__

( self, examples_array, make_doc_fun=None, extra_answer_threshold=3, document_cache=None, training=Tru

examples/longform-qa/eli5_utils.py:325

↓ 1 callersMethod__init__

(self, config)

examples/bert-loses-patience/pabee/modeling_pabee_albert.py:63

↓ 1 callersMethod__init__

(self, config)

examples/bert-loses-patience/pabee/modeling_pabee_bert.py:68

↓ 1 callersMethod__repr__

(self)

templates/adding_a_new_example_script/utils_xxx.py:59

↓ 1 callersMethod__repr__

(self)

examples/contrib/run_swag.py:65

↓ 1 callersMethod_accumulate_gradients

Accumulates the gradients across all the replica.

src/transformers/trainer_tf.py:438

↓ 1 callersMethod_accumulate_next_gradients

Accumulates the gradients from the next element in dataset.

src/transformers/trainer_tf.py:422

↓ 1 callersMethod_add_dataclass_arguments

(self, dtype: DataClassType)

src/transformers/hf_argparser.py:42

↓ 1 callersFunction_add_missing_period

(line)

examples/seq2seq/bertabs/utils_summarization.py:96

↓ 1 callersMethod_add_tokens

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

src/transformers/tokenization_utils.py:185

↓ 1 callersMethod_apply_gradients

Applies the gradients (cross-replica).

src/transformers/trainer_tf.py:407

↓ 1 callersMethod_attend

( self, query_vectors, key_vectors, value_vectors, sorted_bucket_idx_p

src/transformers/modeling_reformer.py:578

↓ 1 callersMethod_attn

(self, inputs, training=False)

src/transformers/modeling_tf_openai.py:104

↓ 1 callersMethod_attn

(self, inputs, training=False)

src/transformers/modeling_tf_gpt2.py:97

↓ 1 callersMethod_attn

(self, q, k, v, attention_mask=None, head_mask=None, output_attentions=False)

src/transformers/modeling_gpt2.py:146

↓ 1 callersMethod_attn

(self, q, k, v, attention_mask=None, head_mask=None, output_attentions=False)

src/transformers/modeling_openai.py:163

↓ 1 callersMethod_batch_encode_plus

( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], Lis

src/transformers/tokenization_utils_fast.py:284

↓ 1 callersMethod_batch_encode_plus

( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInp

src/transformers/tokenization_utils_base.py:1834

↓ 1 callersMethod_batch_prepare_for_model

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates

src/transformers/tokenization_utils.py:560

↓ 1 callersMethod_build_from_file

(self, vocab_file)

src/transformers/tokenization_transfo_xl.py:174

↓ 1 callersFunction_cast_yaml_str

(v)

src/transformers/convert_marian_to_pytorch.py:64

↓ 1 callersMethod_cat_prev_key_padding_mask

( key_padding_mask: Optional[Tensor], prev_key_padding_mask: Optional[Tensor], batch_s

src/transformers/modeling_bart.py:703

↓ 1 callersFunction_check_is_max_context

Check if this is the 'max context' doc span for the token.

templates/adding_a_new_example_script/utils_xxx.py:442

↓ 1 callersMethod_check_layer_entries

(self)

src/transformers/convert_marian_to_pytorch.py:420

↓ 1 callersMethod_chunk

convert into overlapping chunkings. Chunk size = 2w, overlap size = w

src/transformers/modeling_longformer.py:138

↓ 1 callersMethod_compile_space_around_punctuation_pattern

(self)

src/transformers/tokenization_transfo_xl.py:142

↓ 1 callersMethod_compute_attn_mask

(self, query_indices, key_indices, attention_mask, sequence_length)

src/transformers/modeling_reformer.py:672

↓ 1 callersMethod_compute_attn_mask

(self, query_indices, key_indices, attention_mask, query_key_dots_shape, sequence_length)

src/transformers/modeling_reformer.py:925

↓ 1 callersMethod_convert_encoding

Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict. Overflowing tokens are converted to

src/transformers/tokenization_utils_fast.py:143

↓ 1 callersMethod_convert_head_mask_to_5d

-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]

src/transformers/modeling_utils.py:252

↓ 1 callersMethod_convert_id_to_token

(self, index: int)

src/transformers/tokenization_utils.py:862

↓ 1 callersMethod_convert_token_to_id

(self, token)

src/transformers/tokenization_utils.py:399

↓ 1 callersMethod_convert_token_to_id_with_added_voc

(self, token: str)

src/transformers/tokenization_utils_fast.py:203

↓ 1 callersMethod_convert_token_to_id_with_added_voc

(self, token)

src/transformers/tokenization_utils.py:391

↓ 1 callersMethod_do_use_weight_decay

Whether to use L2 weight decay for `param_name`.

src/transformers/optimization_tf.py:181

↓ 1 callersMethod_embedding

Applies embedding based on inputs tensor.

src/transformers/modeling_tf_electra.py:96

↓ 1 callersMethod_embedding

Applies embedding based on inputs tensor.

src/transformers/modeling_tf_albert.py:123

↓ 1 callersMethod_embedding

Applies embedding based on inputs tensor.

src/transformers/modeling_tf_mobilebert.py:154

↓ 1 callersMethod_embedding

Applies embedding based on inputs tensor.

src/transformers/modeling_tf_utils.py:1668

↓ 1 callersMethod_embedding

Applies embedding based on inputs tensor.

src/transformers/modeling_tf_bert.py:172

↓ 1 callersMethod_embedding

Parameters ---------- input_ids: tf.Tensor(bs, max_seq_length) The token ids to embed. Outputs -

src/transformers/modeling_tf_distilbert.py:138

↓ 1 callersMethod_encode_plus

( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[U

src/transformers/tokenization_utils_base.py:1739

↓ 1 callersMethod_evaluate_steps

One step evaluation across replica. Args: per_replica_features: the batched features. per_replica_labels: the bat

src/transformers/trainer_tf.py:161

↓ 1 callersMethod_fast_translate_batch

Beam Search using the encoder inputs contained in `batch`.

examples/seq2seq/bertabs/modeling_bertabs.py:800

↓ 1 callersFunction_filter_out_falsey_values

Remove entries that are None or [] from an iterable.

src/transformers/modeling_bart.py:795

← previous next →501–600 of 4,119, ranked by callers