MCPcopy Create free account

hub / github.com/databrickslabs/dolly / functions

Functions20 in github.com/databrickslabs/dolly

⨍ Functions 20◇ Types & classes 2↳ Endpoints 1
↓ 4 callersFunctionload_model_tokenizer_for_generate
Loads the model and tokenizer so that it can be used for generating responses. Args: pretrained_model_name_or_path (str): name or path fo
training/generate.py:25
↓ 2 callersFunctiongenerate_response
Given an instruction, uses the model and tokenizer to generate a response. This formats the instruction in the instruction format that the model
training/generate.py:220
↓ 2 callersFunctionget_special_token_id
Gets the token ID for a given string that has been added to the tokenizer as a special token. When training, we configure the tokenizer so that t
training/generate.py:43
↓ 2 callersFunctionload_tokenizer
(pretrained_model_name_or_path: str = DEFAULT_INPUT_MODEL)
training/trainer.py:120
↓ 2 callersFunctionload_training_dataset
(path_or_dataset: str = DEFAULT_TRAINING_DATASET)
training/trainer.py:88
↓ 1 callersFunctionget_model_tokenizer
( pretrained_model_name_or_path: str = DEFAULT_INPUT_MODEL, *, gradient_checkpointing: bool = False )
training/trainer.py:138
↓ 1 callersFunctionload_model
( pretrained_model_name_or_path: str = DEFAULT_INPUT_MODEL, *, gradient_checkpointing: bool = False )
training/trainer.py:128
↓ 1 callersFunctionmain
(**kwargs)
training/trainer.py:323
↓ 1 callersFunctionpreprocess_dataset
Loads the training dataset and tokenizes it so it is ready for training. Args: tokenizer (AutoTokenizer): Tokenizer tied to the model.
training/trainer.py:148
↓ 1 callersFunctiontrain
( *, input_model: str, local_output_dir: str, dbfs_output_dir: str, epochs: int, per_d
training/trainer.py:182
Method__init__
Initialize the pipeline Args: do_sample (bool, optional): Whether or not to use sampling. Defaults to True. max_new_t
training/generate.py:66
Function_add_text
(rec)
training/trainer.py:93
Method_forward
(self, model_inputs, **generate_kwargs)
training/generate.py:126
Method_sanitize_parameters
(self, return_full_text: bool = None, **generate_kwa
training/generate.py:82
Methodpostprocess
(self, model_outputs, response_key_token_id, end_key_token_id, return_full_text: bool = False)
training/generate.py:153
Methodpreprocess
(self, instruction_text, **generate_kwargs)
training/generate.py:116
Functionpreprocess_batch
(batch: Dict[str, List], tokenizer: AutoTokenizer, max_length: int)
training/trainer.py:80
Functiontest_load_training_dataset
Make sure we can load the training dataset and it has records
test/test_trainer.py:8
Functiontest_tokenizer
Make sure we can encode and decode with the tokenizer
test/test_trainer.py:3
Methodtorch_call
(self, examples: List[Union[List[int], Any, Dict[str, Any]]])
training/trainer.py:49