Performs all reduce of the tensors after fusing and sparsification. Args: tensor(List of Tensors): a list of tensors to be all-reduced accumulation(Tensor): local gradient accumulation spars(float): a parameter to control sparsity as defined b
(self, tensor, accumulation, spars, topK)
| 801 | self.communicator.sparsification(tensor, accumulation, spars, topK) |
| 802 | |
| 803 | def fused_sparsification(self, tensor, accumulation, spars, topK): |
| 804 | """Performs all reduce of the tensors after fusing and sparsification. |
| 805 | |
| 806 | Args: |
| 807 | tensor(List of Tensors): a list of tensors to be all-reduced |
| 808 | accumulation(Tensor): local gradient accumulation |
| 809 | spars(float): a parameter to control sparsity as defined below |
| 810 | topK(bool): When topK is False, it sparsifies the gradient with absolute |
| 811 | value >= sparsWhen topK is True, it sparsifies a fraction of total gradient |
| 812 | number equals to spars, E.g. when spars = 0.01, it sparsifies 1 % of the |
| 813 | total gradient elements |
| 814 | """ |
| 815 | tensor = singa.VecTensor(tensor) |
| 816 | if accumulation is None: |
| 817 | self.communicator.fusedSparsification(tensor, spars, topK) |
| 818 | else: |
| 819 | self.communicator.fusedSparsification(tensor, accumulation, spars, |
| 820 | topK) |
| 821 | |
| 822 | def wait(self): |
| 823 | """Wait for the cuda streams used by the communicator to finish their operations.""" |
no test coverage detected