↓ 2 callersFunctionprepare_4d_attention_maskr"""Expand 2d attention mask to 4d attention mask. Expand the attention mask with indices from (batch_size, seq_len) to (batch_size, 1, seq_len,
方案分享/赛题三/ljq13317046284/代码/LLaMA-Factory-main/src/llamafactory/data/collator.py:41
↓ 2 callersFunctionrepeat_kv This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he
方案分享/赛题三/双人成行/code/RL/verl/verl/models/llama/megatron/layers/parallel_attention.py:163