Functions — github.com/OpenBMB/CPM-Bee

CodeHub MCPcopy Create free account

hub / github.com/OpenBMB/CPM-Bee / functions

Functions302 in github.com/OpenBMB/CPM-Bee

⨍ Functions 302◇ Types & classes 69

src/cpm_live/tokenizers/ant.py:132

src/cpm_live/tokenizers/bee.py:104

src/cpm_live/dataset/distributed_dataset.py:613

Number of hypotheses in the list.

src/cpm_live/generation/generation_utils.py:81

src/cpm_live/training_tasks/bee/pretrain.py:278

src/cpm_live/training_tasks/ant/pretrain.py:28

src/cpm_live/tokenizers/bee.py:47

Method_convert_to_tensors

(self, input_text, task_id=2)

src/cpm_live/generation/ant.py:14

Beam search Args: model_inputs (dict): input ids. beam_size (int, optional, defaults to 3): beam size of beam

src/cpm_live/generation/ant.py:52

Top-k and top-p sampling. Args: model_inputs (dict): input ids generate_length (int, optional, defaults to 10

src/cpm_live/generation/ant.py:279

Beam search Args: model_inputs (dict): input ids. beam_size (int, optional, defaults to 3): beam size of beam

src/cpm_live/generation/bee.py:255

Method_empty_transform_func

(data: CPMBeeInputType, num_sample: int, r: random.Random)

src/cpm_live/training_tasks/bee/pretrain.py:394

Method_expand_mapping

( data: CPMBeeInputType, stars: List[str], path: List[str], target: List[str] )

src/cpm_live/training_tasks/bee/pretrain.py:309

Function_mixed_dataset_process

( config_path: str, q_cmd: multiprocessing.Queue, q_cmd_out: multiprocessing.Queue, q_data: mu

src/cpm_live/training_tasks/bee/pretrain.py:742

Method_put_placeholder

(data: Any, path: List[str] = [])

src/cpm_live/generation/bee.py:21

Method_transform

(data: CPMBeeInputType)

src/cpm_live/training_tasks/bee/pretrain.py:426

Method_walk_transform_dict

(data: Union[Dict[str, Any], str], prefix: str = "")

src/cpm_live/training_tasks/bee/pretrain.py:294

Functionbmt_save

(file_path: str, model: CPMBee, export_files: Optional[List[str]] = None)

src/cpm_live/utils/export.py:38

src/cpm_live/tokenizers/ant.py:105

src/cpm_live/tokenizers/ant.py:117

src/cpm_live/tokenizers/bee.py:93

src/cpm_live/tokenizers/ant.py:152

src/cpm_live/dataset/distributed_dataset.py:673

src/cpm_live/training_tasks/bee/pretrain.py:979

Functioncompact_dataset

Compact the dataset, removes blocks which the files were deleted. **Note** This may affect the existing dataset state dict. Args: pa

src/cpm_live/dataset/utils.py:128

Methodconvert_ids_to_tokens

src/cpm_live/tokenizers/ant.py:158

Methodconvert_tokens_to_ids

src/cpm_live/tokenizers/ant.py:155

src/cpm_live/training_tasks/ant/pretrain.py:32

Decode ids into a string.

src/cpm_live/tokenizers/bee.py:208

Methoddeserialize

(self, data: bytes)

src/cpm_live/dataset/serializer.py:38

Methoddeserialize

(self, data: bytes)

src/cpm_live/dataset/serializer.py:49

Methoddeserialize

(self, data: bytes)

src/cpm_live/dataset/serializer.py:60

( self, text: str, past_table: Dict[int, str] = {} )

src/cpm_live/tokenizers/bee.py:186

src/cpm_live/tokenizers/ant.py:109

src/cpm_live/tokenizers/ant.py:113

src/cpm_live/tokenizers/bee.py:89

(ctx, x: torch.Tensor, alpha: float)

src/cpm_live/utils/gradient_shrink.py:6

Args: x (:obj:`torch.Tensor` of shape ``(batch_size, seq_len, dim_norm)``): Input tensor that need to be normalized. Retu

src/cpm_live/layers/layernorm.py:30

Args: hidden_states (:obj:`torch.Tensor` of shape ``(batch, seq_self, dim_model)``): Input of self-attention block. It can be the

src/cpm_live/layers/blocks.py:67

Args: hidden_states (:obj:`torch.Tensor` of shape ``(batch, seq_self, dim_model)``): Hidden states before feed forward layer.

src/cpm_live/layers/blocks.py:141

Args: self_hidden_states (:obj:`torch.Tensor` of shape ``(batch, seq_self, dim_model)``): Input of transformer block(self-attenti

src/cpm_live/layers/blocks.py:209

Args: x (:obj:`torch.Tensor` of shape ``(batch, seq_len, dim_in)``): The input of linear layer Returns: :obj:

src/cpm_live/layers/linear.py:44

( self, key_pos: torch.Tensor, query_pos: torch.Tensor, key_segment: torch.Ten

src/cpm_live/layers/position_embedding.py:50

( self, query_pos: torch.Tensor, # (batch, len_q) key_pos: torch.Tensor, # (batch, l

src/cpm_live/layers/position_embedding.py:156

Args: x (:obj:`torch.Tensor` of shape ``(..., dim)``): Inputs. x_pos (:obj:`torch.Tensor` of shape ``(...)``): Positi

src/cpm_live/layers/position_embedding.py:236

This model inherits from bmt.DistributedModule. Args: hidden_q (:obj:`torch.Tensor` of shape ``(batch, len_q, dim_model)``): Indic

src/cpm_live/layers/attention.py:52

Args: hidden-states (:obj:`torch.Tensor` of shape ``(batch, seq_enc, dim_model)``): Input of encoder, might be the embedding of a

src/cpm_live/layers/transformer.py:87

Args: ids (:obj:`torch.Tensor` of shape ``(batch_size, seq_len)``): Indices of input sequence tokens. Return:

src/cpm_live/layers/embedding.py:44

Args: ids (:obj:`torch.Tensor` of shape ``(batch_size, seq_len)``): Indices of input sequence tokens. ids (:obj:`torc

src/cpm_live/layers/embedding.py:92

This model inherits from bmt.DistributedModule. Transform an input tensor from one feature space to another via a nonlinear operation

src/cpm_live/layers/feedforward.py:46

Args: x (:obj:`torch.Tensor` of shape ``(batch, seq_len, dim_in)``): The input of feed-forward module. Return:

src/cpm_live/layers/feedforward.py:107

( self, input: torch.Tensor, # (batch, seqlen) length: torch.Tensor, # (batch)

src/cpm_live/models/ant.py:114

( self, input: torch.Tensor, # (batch, seqlen) int32 input_sub: torch.Tensor, # (bat

src/cpm_live/models/bee_torch.py:55

( self, input: torch.Tensor, # (batch, seqlen) length: torch.Tensor, # (batch)

src/cpm_live/models/ant_torch.py:69

( self, input: torch.Tensor, # (batch, seqlen) int32 input_sub: torch.Tensor, # (bat

src/cpm_live/models/bee.py:104

Args: x (:obj:`torch.Tensor` of shape ``(batch_size, seq_len, dim_norm)``): Input tensor that need to be normalized. Retu

src/cpm_live/native_layers/layernorm.py:29

Args: hidden_states (:obj:`torch.Tensor` of shape ``(batch, seq_self, dim_model)``): Input of self-attention block. It can be the

src/cpm_live/native_layers/blocks.py:66

Args: hidden_states (:obj:`torch.Tensor` of shape ``(batch, seq_self, dim_model)``): Hidden states before feed forward layer.

src/cpm_live/native_layers/blocks.py:140

Args: self_hidden_states (:obj:`torch.Tensor` of shape ``(batch, seq_self, dim_model)``): Input of transformer block(self-attenti

src/cpm_live/native_layers/blocks.py:208

Args: x (:obj:`torch.Tensor` of shape ``(batch, seq_len, dim_in)``): The input of linear layer Returns: :obj:

src/cpm_live/native_layers/linear.py:38

( self, key_pos: torch.Tensor, query_pos: torch.Tensor, key_segment: torch.Ten

src/cpm_live/native_layers/position_embedding.py:46

( self, query_pos: torch.Tensor, # (batch, len_q) key_pos: torch.Tensor, # (batch, l

src/cpm_live/native_layers/position_embedding.py:149

Args: x (:obj:`torch.Tensor` of shape ``(..., dim)``): Inputs. x_pos (:obj:`torch.Tensor` of shape ``(...)``): Positi

src/cpm_live/native_layers/position_embedding.py:229

Args: hidden_q (:obj:`torch.Tensor` of shape ``(batch, len_q, dim_model)``): Indices of input sequence tokens. It will be embedde

src/cpm_live/native_layers/attention.py:51

Args: hidden-states (:obj:`torch.Tensor` of shape ``(batch, seq_enc, dim_model)``): Input of encoder, might be the embedding of a

src/cpm_live/native_layers/transformer.py:84

Args: ids (:obj:`torch.Tensor` of shape ``(batch_size, seq_len)``): Indices of input sequence tokens. Return:

src/cpm_live/native_layers/embedding.py:40

Args: ids (:obj:`torch.Tensor` of shape ``(batch_size, seq_len)``): Indices of input sequence tokens. ids (:obj:`torc

src/cpm_live/native_layers/embedding.py:85

Transform an input tensor from one feature space to another via a nonlinear operation Args: x (:obj:`torch.Tensor` of shape ``(ba

src/cpm_live/native_layers/feedforward.py:45

Args: x (:obj:`torch.Tensor` of shape ``(batch, seq_len, dim_in)``): The input of feed-forward module. Return:

src/cpm_live/native_layers/feedforward.py:105

(self, data_list, **kwargs)

src/cpm_live/generation/bee.py:214

Methodinference

( self, input: torch.Tensor, # (batch, len_q) int32 input_sub: torch.Tensor, # (batc

src/cpm_live/models/bee_torch.py:137

Methodinference

( self, input: torch.Tensor, # (batch, seqlen) length: torch.Tensor, # (batch)

src/cpm_live/models/ant_torch.py:111

Methodinference

( self, input: torch.Tensor, # (batch, len_q) int32 input_sub: torch.Tensor, # (batc

src/cpm_live/models/bee.py:186

Functionjson_save

(file_path: str, obj: object, export_files: Optional[List[str]] = None)

src/cpm_live/utils/export.py:52

Methodload_state_dict

Load dataset state. Args: state (dict): dataset state dict. strict (bool): If `strict` is True, world size needs to b

src/cpm_live/dataset/distributed_dataset.py:467

Functionmask_dataset

Mask one file in dataset. Blocks in masked datasets won't be read later. Args: path (str): path to dataset dbname (str): file nam

src/cpm_live/dataset/utils.py:173

src/cpm_live/tokenizers/bee.py:101

Functionmerge_dataset

(dst: str, src: str)

src/cpm_live/dataset/utils.py:200

src/cpm_live/dataset/distributed_dataset.py:666

src/cpm_live/dataset/distributed_dataset.py:584

src/cpm_live/dataset/distributed_dataset.py:662

Methodnewline_id

src/cpm_live/tokenizers/ant.py:129

src/cpm_live/dataset/distributed_dataset.py:670

src/cpm_live/tokenizers/ant.py:121

Methodprojection

Projection based on embedding's weight. For example, embedding map vocab_size to embed_size, than projection map embed_size back to vocab_siz

src/cpm_live/layers/embedding.py:104

Methodprojection

Projection based on embedding's weight. For example, embedding map vocab_size to embed_size, than projection map embed_size back to vocab_siz

src/cpm_live/native_layers/embedding.py:51

Methodprojection

Projection based on embedding's weight. For example, embedding map vocab_size to embed_size, than projection map embed_size back to vocab_siz

src/cpm_live/native_layers/embedding.py:97

Read a piece of data from dataset. Workers in different ranks will read different data.

src/cpm_live/dataset/distributed_dataset.py:548

Functionrename_if_exists_decorator

src/cpm_live/utils/export.py:29

Methodserialize

src/cpm_live/dataset/serializer.py:35

Methodserialize

src/cpm_live/dataset/serializer.py:46

Methodserialize

src/cpm_live/dataset/serializer.py:57

Methodstate_dict

Returns a state dict representing the read states of the dataset. Example: >>> state = dataset.state_dict() >>> datas

src/cpm_live/dataset/distributed_dataset.py:419

Methodto_json_file

(self, json_file_path: Union[str, os.PathLike])

src/cpm_live/utils/config.py:67

src/cpm_live/tokenizers/ant.py:41

Functiontorch_save

(file_path: str, obj: object, export_files: Optional[List[str]] = None)

src/cpm_live/utils/export.py:45

src/cpm_live/tokenizers/bee.py:183

src/cpm_live/tokenizers/ant.py:125

src/cpm_live/tokenizers/bee.py:97

Methodvocab_size

src/cpm_live/tokenizers/ant.py:101

← previous next →201–300 of 302, ranked by callers