Prediction layer (cross_entropy or adaptive_softmax).

Same as https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py, without amsgrad, with step in a tensor, and states initialization

Assign LR based on a cyclical schedule that follows the cosine function. See https://arxiv.org/pdf/1608.03983.pdf for details. We also su

Decay the LR based on the inverse square root of the update number. We also support a warmup phase where we linearly increase the learning ra