Performs all reduce of a tensor after sparsification. Args: tensor(Tensor): a tensor to be all-reduced accumulation(Tensor): local gradient accumulation spars(float): a parameter to control sparsity as defined below topK(bool):
(self, tensor, accumulation, spars, topK)
| 784 | self.communicator.fusedSynchHalf(tensor, send) |
| 785 | |
| 786 | def sparsification(self, tensor, accumulation, spars, topK): |
| 787 | """Performs all reduce of a tensor after sparsification. |
| 788 | |
| 789 | Args: |
| 790 | tensor(Tensor): a tensor to be all-reduced |
| 791 | accumulation(Tensor): local gradient accumulation |
| 792 | spars(float): a parameter to control sparsity as defined below |
| 793 | topK(bool): When topK is False, it sparsifies the gradient with absolute |
| 794 | value >= sparsWhen topK is True, it sparsifies a fraction of total gradient |
| 795 | number equals to spars, E.g. when spars = 0.01, it sparsifies 1 % of the |
| 796 | total gradient elements |
| 797 | """ |
| 798 | if accumulation is None: |
| 799 | self.communicator.sparsification(tensor, spars, topK) |
| 800 | else: |
| 801 | self.communicator.sparsification(tensor, accumulation, spars, topK) |
| 802 | |
| 803 | def fused_sparsification(self, tensor, accumulation, spars, topK): |
| 804 | """Performs all reduce of the tensors after fusing and sparsification. |
no outgoing calls
no test coverage detected