MCPcopy Index your code
hub / github.com/pytorch/examples / sequence_parallel_example.py

File sequence_parallel_example.py

distributed/tensor_parallelism/sequence_parallel_example.py:None–None  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1"""
2This is the script to test Sequence Parallel(SP) on a toy model in a
3Megetron-LM SPMD style. We show an E2E working flow from forward,
4backward and optimization.

Callers

nothing calls this directly

Calls 4

verify_min_gpu_countFunction · 0.90
get_loggerFunction · 0.90
rank_logFunction · 0.90
ToyModelClass · 0.70

Tested by

no test coverage detected