Updates the loss_scaling. Args: new_scale(float, optional): New loss scaling factor. If provided, the loss scaling facto

Get current worker index. Returns: int: node id

Used for Construct std::vector<Tensor> name

Generates instructions to load a constant value.

Get test results from doctester with docstrings to test.

Copy parameters and buffers from :attr:`state_dict` into this module and its descendants. If :attr:`strict` is ``True``, then

Message for logging with api `count` and running `env`.

Add a IShapeLayer to get the shape of `input` ITensor. This includes a workaround that casting the shape result(int64) from TRT10 back to int

We use the set 'neglected_output' here, because some Ops like batch norm, the outputs specified in the op des are only used during training, so we sho

* Generate flag used to create event on all sorts of equipment. * NOTE: Support CPU/CUDA/ROCM currently. */

Convert the attention mask to the target dtype we expect. Parameters: attn_mask (Tensor, optional): A tensor used in multi-head atte

Given a rank and the processes mesh the rank belongs to, compute the communication peers of the rank based on the give axis in the mesh.

r""" Transform ratio within [0, 1] to percentage presentation.

获得pp_idx的mesh

Check whether the node is the first instance of worker. Returns: bool: True if this is the first node of worker,

Pad tensor according to ``'pad'`` and ``'mode'``. Note: 1. Denote ``'x'``'s dimension as N (same in the following). If mode is ``'co

Wait until parameter servers are ready, use connext_ex to detect port readiness. Args: endpoints (list|tuple): endpoints string list