Performs all reduce of the tensors after fusing them in a buffer. Args: tensor(List of Tensors): a list of tensors to be all-reduced send(bool): When send is False, the tensor won't be send to the target device immediately, it will be copied t
(self, tensor, send=True)
| 754 | self.communicator.synch(tensor) |
| 755 | |
| 756 | def fused_all_reduce(self, tensor, send=True): |
| 757 | """Performs all reduce of the tensors after fusing them in a buffer. |
| 758 | |
| 759 | Args: |
| 760 | tensor(List of Tensors): a list of tensors to be all-reduced |
| 761 | send(bool): When send is False, the tensor won't be send to the |
| 762 | target device immediately, it will be copied to the buffer first |
| 763 | """ |
| 764 | tensor = singa.VecTensor(tensor) |
| 765 | self.communicator.fusedSynch(tensor, send) |
| 766 | |
| 767 | def all_reduce_half(self, tensor): |
| 768 | """Performs all reduce of a tensor after converting to FP16. |