Disables the adapter module.

Disable adapter layers by swapping out AdaptedAttention modules.

Enable adapter layers by swapping in cached AdaptedAttention modules.

Input shape: Batch x Time x Channel attention_mask: [bsz, q_len]

qkv: (batch, seqlen, 3, nheads, headdim) bias: optional, shape broadcastible to (batch, nheads, seqlen, seqlen).

q: (batch, seqlen_q, nheads, headdim) kv: (batch, seqlen_k, 2, nheads, headdim) bias: optional, shape broadcastib

q: (batch_size, seqlen_q, nheads, headdim) k, v: (batch_size, seqlen_k, nheads, headdim) bias: optional, shape br

Forward pass of the model.

Forward pass for the adapter which wraps the original LlamaAttention module. "Official" paper implementation: https://github

Returns a Peft config object from a dictionary. Args: config_dict (`Dict[str, Any]`): Dictionary containing the configuration parame

Compute query states for Llama models specifically. They need to be recomputed as the forward() method of the original LlamaModel in the tra

r""" This method merges the LoRa layers into the base model. This is needed if someone wants to use the base model as a standalone mod

Perform TaSL consolidation for a specific task Args: task_id: ID of the current task output_dir: Directory where models and

r""" This method wraps the entire protocol for preparing a model before running a training. This includes: 1- Cast the layernorm in fp32 2

Prints the number of trainable parameters in the model.

Recursively replace all SiLU activation functions with ReLU in the model.

Save importance scores to a CSV file