存储不含有显卡信息的state_dict或model :param model: :param model_name: :param only_param: :return:

保存成pickle文件 :param data: :param file_name: :param pickle_path: :return:

Save a configuration object to the directory `save_directory`, so that it can be re-loaded using the :func:`~transformers.PretrainedConfi

Save the tokenizer vocabulary files together with: - added tokens, - special-tokens-to-class-attributes-mapping,

Save the tokenizer vocabulary and merge files to a directory.

Save the tokenizer vocabulary and merge files to a directory.

Save the tokenizer vocabulary to a directory or file.

Save the tokenizer vocabulary and merge files to a directory.

Save the sentencepiece vocabulary (copy original file) and special tokens file to a directory.

Save the tokenizer vocabulary to a directory or file.

Separation token (string). E.g. separate context and query in an input sequence. Log an error if used while not having been set.

Id of the separation token in the vocabulary. E.g. separate context and query in an input sequence. Log an error if used while not having been set.

separate heads

projection

A dictionary mapping special token class attribute (cls_token, unk_token...) to their values ('<unk>', '<cls>'...)

打印模型结构信息 :param model: :param inputs: :param batch_size: :param show_input: :return: Example: >>> print("

Serializes this instance to a JSON string.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi

Unknown token (string). Log an error if used while not having been set.

Id of the unknown token in the vocabulary. Log an error if used while not having been set.

group heads

compute context

Size of the base vocabulary (without the added tokens)

batch should be a list of (sequence, target, length) tuples... Returns a padded tensor of sequences sorted from longest to shortest,