MCPcopy Create free account
hub / github.com/ckd0817/LLM-Interview-Code / GroupQueryAttention

Class GroupQueryAttention

attention/GroupQueryAttention.py:16–151  ·  view source on GitHub ↗

分组查询注意力模块 在 GQA 中: - Q 有 num_heads 个头 - K 和 V 只有 num_kv_heads 个头(num_kv_heads < num_heads) - 多个 Q 头共享同一个 KV 头(通过 repeat_kv 实现) 优势: - 比 MHA 更少的 KV 缓存,提高推理效率 - 比 MQA 更好的性能,保持模型质量 Args: model_dim: 模型隐藏维度 num_heads: 查询头数 num_kv_heads: 键值头数(

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected