Stride between real and imaginary parts

Loads a fragment from memory with additional logical offset

Loads a fragment from memory with additional logical offset

Possibly convert HWIO to OIHW.

Computes padding in shared memory to perform efficient transpose without bank conflicts.

Returns the number of logical elements

Returns the stride of the layout

Returns the stride of the layout

Helper to rematerialize block Idx. Reduces register liveness.

Reads a tab separated value file.

Reads a tab separated value file.

Adds a pointer offset in units of Element

Returns the number of scalar elements needed to store tensor.

convert TensorFlow Tensor to FasterTransformer Tensor

Returns a list of the shape of tensor, preferring static dimensions. Args: tensor: A tf.Tensor object to find the shape of. expected_rank:

Get the tensor model parallel group the caller rank belongs to.

Return my rank for the tensor model parallel group.

Return world size for the tensor model parallel group.

Returns ImplicitGemm tensor C size in number of elements

Init method based on N(0, sigma).

inputs: logits: [batch_size, vocab_size], the values of log logits threshold: int when using top_k, and a probabi

Overrides the internal iteration index

Stores a fragment to memory at the location pointed to by the iterator

Stores a fragment to memory with additional pointer offset

Computes laneId within a warp

Whether to include special tokens in encoded text. This should be determined by whether or not the model was trained with special tokens.

Increment and return an instance to self.

Reduce a tensor of losses across all GPUs.

Determines whether the Implicit GEMM can execute the given problem.

* Checks whether a flag "--<flag>" is present in the commandline */

Creates a `truncated_normal_initializer` with the given range.

Creates a `truncated_normal_initializer` with the given range.

Helper method to get the model chunk ID given the iteration number.

Return world size for the pipeline model parallel group.

Loads a fragment from memory with additional logical offset

Helper returns a layout to a tightly packed tensor

Accesses raw internal state

Executes one test

Return a view of the list of tokens from [i, j).

Communicate tensors between stages. Used as helper method in other communication methods that are used in megatron/schedules.py. Takes the fo

Insert a new compiled device module

Adds a pointer offset in units of Element

Adds a tile offset in the unit of tile.

Call this method to enable sparsity. If init(...) was called with allow_recompute_mask=False AND sparsity is disabled, pruned field can be Non

Converts a sequence of tokens into ids using the vocab.

Convert cutlass status to status strings

Returns a list of the shape of tensor, preferring static dimensions. Args: tensor: A tf.Tensor object to find the shape of. expected_rank:

Returns a list of the shape of tensor, preferring static dimensions. Args: tensor: A tf.Tensor object to find the shape of. expected_rank:

Initializes data structures

Least positive value

If distributed is initialized, print only on last rank.