Per-rank worker: build, capture, profile, and (on rank 0) post-process.

r""" Each trainer runs a forward pass which involves an embedding lookup on the parameter server and running nn.Linear locally. During the bac

Spawn a ``world_size=2`` group and surface the comm metadata.

The conditional statement (aka predicate) passed to ``cond()`` must be one of the following: - ``torch.Tensor`` with a single element - b

Trim everything except the function name.

Forward pass with annotated regions.

In the forward function we accept a Tensor of input data and we must return a Tensor of output data. We can use Modules defined in th

For the forward pass of the model, we randomly choose either 4, 5 and reuse the e parameter to compute the contribution of these orde