Linear layer with column parallelism. The linear layer is defined as Y = XA + b. A is parallelized along its second dimension as A = [A_1, ..., A_p]. Arguments: in_features: first dimension of matrix A. out_features: second dimension of matrix A. bias: If true,
| 56 | |
| 57 | |
| 58 | class LoraColumnParallelLinear(ColumnParallelLinear): |
| 59 | """Linear layer with column parallelism. |
| 60 | |
| 61 | The linear layer is defined as Y = XA + b. A is parallelized along |
| 62 | its second dimension as A = [A_1, ..., A_p]. |
| 63 | |
| 64 | Arguments: |
| 65 | in_features: first dimension of matrix A. |
| 66 | out_features: second dimension of matrix A. |
| 67 | bias: If true, add bias |
| 68 | gather_output: If true, call all-gether on output and make Y avaiable |
| 69 | to all GPUs, otherwise, every GPU will have its output |
| 70 | which is Y_i = XA_i |
| 71 | init_method: method to initialize weights. Note that bias is always set |
| 72 | to zero. |
| 73 | stride: For the strided linear layers. |
| 74 | keep_master_weight_for_test: This was added for testing and should be |
| 75 | set to False. It returns the master weights |
| 76 | used for initialization. |
| 77 | """ |
| 78 | |
| 79 | def __init__( |
| 80 | self, |
| 81 | in_features: int, |
| 82 | out_features: int, |
| 83 | bias: bool = True, |
| 84 | gather_output: bool = True, |
| 85 | init_method: Callable[[torch.Tensor], torch.Tensor] = init.xavier_normal_, |
| 86 | stride: int = 1, |
| 87 | keep_master_weight_for_test: bool = False, |
| 88 | lora_rank=0 |
| 89 | ) -> None: |
| 90 | nn.Module.__init__(self) |
| 91 | |
| 92 | # Keep input parameters |
| 93 | self.in_features = in_features |
| 94 | self.out_features = out_features |
| 95 | self.gather_output = gather_output |
| 96 | # Divide the weight matrix along the last dimension. |
| 97 | world_size = get_model_parallel_world_size() |
| 98 | self.output_size_per_partition = divide_and_check_no_remainder(out_features, world_size) |
| 99 | |
| 100 | # Parameters. |
| 101 | # Note: torch.nn.functional.linear performs XA^T + b and as a result |
| 102 | # we allocate the transpose. |
| 103 | self.weight = Parameter(torch.Tensor(self.output_size_per_partition, self.in_features)) |
| 104 | if bias: |
| 105 | self.bias = Parameter(torch.Tensor(self.output_size_per_partition)) |
| 106 | # Always initialize bias to zero. |
| 107 | with torch.no_grad(): |
| 108 | self.bias.zero_() |
| 109 | else: |
| 110 | self.register_parameter("bias", None) |
| 111 | |
| 112 | # Initialize weight. |
| 113 | self.master_weight = _initialize_affine_weight( |
| 114 | self.weight, |
| 115 | self.out_features, |