Batch size ramp up. Over steps = (global-batch-size - start-batch-size) / batch_size_increment increment batch size from sta

Args: name: name of the dataset. data_prefix: prefix of the data. documents: list of document indices.

Input optimizer is the base optimizer for example Adam.

Initialize scale value with the input initial scale.

"Grad scaler with dynamic scale that gets adjusted during training.