Performs backward propagation from the loss and parameter update with sparsification. THIS IS A EXPERIMENTAL FUNCTION FOR RESEARCH PURPOSE: From the loss, it performs backward propagation to get the gradients and do the parameter update. It fuses the tensors with size small
(self,
loss,
threshold=2097152,
spars=0.05,
topK=False,
corr=True)
| 992 | self.opt.step() |
| 993 | |
| 994 | def backward_and_sparse_update(self, |
| 995 | loss, |
| 996 | threshold=2097152, |
| 997 | spars=0.05, |
| 998 | topK=False, |
| 999 | corr=True): |
| 1000 | """ Performs backward propagation from the loss and parameter update with sparsification. |
| 1001 | |
| 1002 | THIS IS A EXPERIMENTAL FUNCTION FOR RESEARCH PURPOSE: |
| 1003 | From the loss, it performs backward propagation to get the gradients and do the parameter |
| 1004 | update. It fuses the tensors with size smaller than the threshold value to reduce network |
| 1005 | latency, as well as using sparsification schemes to transfer only the gradient elements which |
| 1006 | are significant. |
| 1007 | |
| 1008 | Args: |
| 1009 | loss(Tensor): loss is the objective function of the deep learning model |
| 1010 | optimization, e.g. for classification problem it can be the output of the |
| 1011 | softmax_cross_entropy function. |
| 1012 | threshold(int): threshold is a parameter to control performance in fusing |
| 1013 | the tensors. For the tensors of sizes smaller than threshold, they are to |
| 1014 | be accumulated and fused before the all reduce operation. For the tensors |
| 1015 | of its size larger than the threshold value, they are to be reduced directly |
| 1016 | without fusion. |
| 1017 | spars(float): a parameter to control sparsity as defined below |
| 1018 | topK(bool): When topK is False, it sparsifies the gradient with absolute |
| 1019 | value >= sparsWhen topK is True, it sparsifies a fraction of total gradient |
| 1020 | number equals to spars, E.g. when spars = 0.01, it sparsifies 1 % of the |
| 1021 | total gradient elements |
| 1022 | corr(bool): whether to use the local accumulate gradient for correction |
| 1023 | |
| 1024 | Attributes: |
| 1025 | self.sparsInit: A counter to determine which partition to perform all-reduce. |
| 1026 | self.gradAccumulation: Local gradient accumulation |
| 1027 | """ |
| 1028 | if ((not hasattr(self, "sparsInit")) and corr): |
| 1029 | self.gradAccumulation = [] |
| 1030 | self.sparsInit = False |
| 1031 | plist = [] |
| 1032 | acc = 0 |
| 1033 | k = -1 |
| 1034 | glist = [] |
| 1035 | for p, g in autograd.backward(loss): |
| 1036 | if g.size() > threshold: |
| 1037 | # larger than threshold -> reduced directly |
| 1038 | k += 1 |
| 1039 | if (corr and (not self.sparsInit)): |
| 1040 | # create a tensor for the gradient accumulation |
| 1041 | flag = p.device.graph_enabled() |
| 1042 | p.device.EnableGraph(False) |
| 1043 | self.gradAccumulation.append( |
| 1044 | tensor.Tensor((g.size(),), p.device, p.dtype)) |
| 1045 | self.gradAccumulation[k].set_value(0.0) |
| 1046 | p.device.EnableGraph(flag) |
| 1047 | if corr: |
| 1048 | self.sparsification(g.data, self.gradAccumulation[k].data, |
| 1049 | spars, topK) |
| 1050 | else: |
| 1051 | self.sparsification(g.data, None, spars, topK) |
no test coverage detected