Measure and return speed performance metrics. This function requires a time snapshot `start_time` before the operation to be measured starts and this function should be run immediately after the operation to be measured has completed. Args: - split: name to prefix metric (like
(split, start_time, num_samples=None, num_steps=None, num_tokens=None)
| 381 | |
| 382 | |
| 383 | def speed_metrics(split, start_time, num_samples=None, num_steps=None, num_tokens=None): |
| 384 | """ |
| 385 | Measure and return speed performance metrics. |
| 386 | |
| 387 | This function requires a time snapshot `start_time` before the operation to be measured starts and this function |
| 388 | should be run immediately after the operation to be measured has completed. |
| 389 | |
| 390 | Args: |
| 391 | - split: name to prefix metric (like train, eval, test...) |
| 392 | - start_time: operation start time |
| 393 | - num_samples: number of samples processed |
| 394 | - num_steps: number of steps processed |
| 395 | - num_tokens: number of tokens processed |
| 396 | """ |
| 397 | runtime = time.time() - start_time |
| 398 | result = {f"{split}_runtime": round(runtime, 4)} |
| 399 | if runtime == 0: |
| 400 | return result |
| 401 | if num_samples is not None: |
| 402 | samples_per_second = num_samples / runtime |
| 403 | result[f"{split}_samples_per_second"] = round(samples_per_second, 3) |
| 404 | if num_steps is not None: |
| 405 | steps_per_second = num_steps / runtime |
| 406 | result[f"{split}_steps_per_second"] = round(steps_per_second, 3) |
| 407 | if num_tokens is not None: |
| 408 | tokens_per_second = num_tokens / runtime |
| 409 | result[f"{split}_tokens_per_second"] = round(tokens_per_second, 3) |
| 410 | return result |
| 411 | |
| 412 | |
| 413 | class SchedulerType(ExplicitEnum): |