origin implement: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Clear GPU memory by running garbage collection and emptying cache.

origin implement: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

origin implement: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

origin implement: down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))

fuse the linear operations in Layernorm into the adjacent linear blocks.

equivalent to had = torch.kron(hadL, hadR) x = x.reshape(-1, had.shape[0]) x = x.matmul(had).reshape(init_shape)

Load CUTLASS operations

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): T

Calculate theoretical GPU memory usage

Check GPU compatibility

fuse the linear operations in Layernorm into the adjacent linear blocks.

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`,

This function takes a linear layer and subtracts the means from the weights and biases. This will result in the linear layer performing t

Benchmark batch FP16 dense GEMM (using PyTorch)

Benchmark batch INT4 dense GEMM

Benchmark batch sparse GEMM