Performs backward propagation and parameter update, with FP16 precision communication. THIS IS A EXPERIMENTAL FUNCTION FOR RESEARCH PURPOSE: From the loss, it performs backward propagation to get the gradients and do the parameter update. For gradient communication, it fuses
(self,
loss,
threshold=2097152,
clipping=False,
clip_Value=100)
| 865 | self.opt.step() |
| 866 | |
| 867 | def backward_and_update_half(self, |
| 868 | loss, |
| 869 | threshold=2097152, |
| 870 | clipping=False, |
| 871 | clip_Value=100): |
| 872 | """Performs backward propagation and parameter update, with FP16 precision communication. |
| 873 | |
| 874 | THIS IS A EXPERIMENTAL FUNCTION FOR RESEARCH PURPOSE: |
| 875 | From the loss, it performs backward propagation to get the gradients and do the parameter |
| 876 | update. For gradient communication, it fuses all the tensor smaller than the threshold value |
| 877 | to reduce network latency, as well as converting them to FP16 half precision format before |
| 878 | sending them out. To assist training, this functions provide an option to perform gradient |
| 879 | clipping. |
| 880 | |
| 881 | Args: |
| 882 | loss(Tensor): loss is the objective function of the deep learning model |
| 883 | optimization, e.g. for classification problem it can be the output of the |
| 884 | softmax_cross_entropy function. |
| 885 | threshold(int): threshold is a parameter to control performance in fusing |
| 886 | the tensors. For the tensors of sizes smaller than threshold, they are to |
| 887 | be accumulated and fused before the all reduce operation. For the tensors |
| 888 | of its size larger than the threshold value, they are to be reduced directly |
| 889 | without fusion. |
| 890 | clipping(bool): a boolean flag to choose whether to clip the gradient value |
| 891 | clip_value(float): the clip value to be used when clipping is True |
| 892 | """ |
| 893 | plist = [] |
| 894 | acc = 0 |
| 895 | glist = [] |
| 896 | for p, g in autograd.backward(loss): |
| 897 | assert p.dtype == tensor.float32, ( |
| 898 | 'This function is only available for input tensor precision 32 bit, ' |
| 899 | 'which are converted into 16 bits before transmit') |
| 900 | if clipping: |
| 901 | g = autograd.clip(g, -clip_Value, clip_Value) |
| 902 | if g.size() > threshold: |
| 903 | # larger than threshold -> reduced directly |
| 904 | self.all_reduce_half(g.data) |
| 905 | else: |
| 906 | # smaller than threshold -> accumulate |
| 907 | glist.append(g.data) |
| 908 | self.fused_all_reduce_half([g.data], send=False) |
| 909 | acc += g.size() |
| 910 | if (acc > threshold): |
| 911 | self.fused_all_reduce_half(glist) |
| 912 | acc = 0 |
| 913 | glist = [] |
| 914 | plist.append((p, g)) |
| 915 | if glist: |
| 916 | self.fused_all_reduce_half(glist) |
| 917 | self.wait() |
| 918 | for p, g in plist: |
| 919 | self.update(p, g) |
| 920 | self.opt.step() |
| 921 | |
| 922 | def backward_and_partial_update(self, loss, threshold=2097152): |
| 923 | """Performs backward propagation from the loss and parameter update using asynchronous training. |
no test coverage detected