Method all_reduce

python/singa/opt.py:748–754

Performs all reduce of a tensor for distributed training. Args: tensor(Tensor): a tensor to be all-reduced

(self, tensor)

        self.opt.update(param, grad)

    def all_reduce(self, tensor):
        """Performs all reduce of a tensor for distributed training.

        Args:
                tensor(Tensor): a tensor to be all-reduced
        """
        self.communicator.synch(tensor)

    def fused_all_reduce(self, tensor, send=True):
        """Performs all reduce of the tensors after fusing them in a buffer.