When the config is defined for certain number of workers (according to ``cfg.SOLVER.REFERENCE_WORLD_SIZE``) that's different from the number of workers currently in use, returns a new cfg where the total batch size is scaled so that the per-GPU batch size stays the s
(cfg, num_workers: int)
| 554 | |
| 555 | @staticmethod |
| 556 | def auto_scale_workers(cfg, num_workers: int): |
| 557 | """ |
| 558 | When the config is defined for certain number of workers (according to |
| 559 | ``cfg.SOLVER.REFERENCE_WORLD_SIZE``) that's different from the number of |
| 560 | workers currently in use, returns a new cfg where the total batch size |
| 561 | is scaled so that the per-GPU batch size stays the same as the |
| 562 | original ``IMS_PER_BATCH // REFERENCE_WORLD_SIZE``. |
| 563 | |
| 564 | Other config options are also scaled accordingly: |
| 565 | * training steps and warmup steps are scaled inverse proportionally. |
| 566 | * learning rate are scaled proportionally, following :paper:`ImageNet in 1h`. |
| 567 | |
| 568 | For example, with the original config like the following: |
| 569 | |
| 570 | .. code-block:: yaml |
| 571 | |
| 572 | IMS_PER_BATCH: 16 |
| 573 | BASE_LR: 0.1 |
| 574 | REFERENCE_WORLD_SIZE: 8 |
| 575 | MAX_ITER: 5000 |
| 576 | STEPS: (4000,) |
| 577 | CHECKPOINT_PERIOD: 1000 |
| 578 | |
| 579 | When this config is used on 16 GPUs instead of the reference number 8, |
| 580 | calling this method will return a new config with: |
| 581 | |
| 582 | .. code-block:: yaml |
| 583 | |
| 584 | IMS_PER_BATCH: 32 |
| 585 | BASE_LR: 0.2 |
| 586 | REFERENCE_WORLD_SIZE: 16 |
| 587 | MAX_ITER: 2500 |
| 588 | STEPS: (2000,) |
| 589 | CHECKPOINT_PERIOD: 500 |
| 590 | |
| 591 | Note that both the original config and this new config can be trained on 16 GPUs. |
| 592 | It's up to user whether to enable this feature (by setting ``REFERENCE_WORLD_SIZE``). |
| 593 | |
| 594 | Returns: |
| 595 | CfgNode: a new config. Same as original if ``cfg.SOLVER.REFERENCE_WORLD_SIZE==0``. |
| 596 | """ |
| 597 | old_world_size = cfg.SOLVER.REFERENCE_WORLD_SIZE |
| 598 | if old_world_size == 0 or old_world_size == num_workers: |
| 599 | return cfg |
| 600 | cfg = cfg.clone() |
| 601 | frozen = cfg.is_frozen() |
| 602 | cfg.defrost() |
| 603 | |
| 604 | assert ( |
| 605 | cfg.SOLVER.IMS_PER_BATCH % old_world_size == 0 |
| 606 | ), "Invalid REFERENCE_WORLD_SIZE in config!" |
| 607 | scale = num_workers / old_world_size |
| 608 | bs = cfg.SOLVER.IMS_PER_BATCH = int(round(cfg.SOLVER.IMS_PER_BATCH * scale)) |
| 609 | lr = cfg.SOLVER.BASE_LR = cfg.SOLVER.BASE_LR * scale |
| 610 | max_iter = cfg.SOLVER.MAX_ITER = int(round(cfg.SOLVER.MAX_ITER / scale)) |
| 611 | warmup_iter = cfg.SOLVER.WARMUP_ITERS = int(round(cfg.SOLVER.WARMUP_ITERS / scale)) |
| 612 | cfg.SOLVER.STEPS = tuple(int(round(s / scale)) for s in cfg.SOLVER.STEPS) |
| 613 | cfg.TEST.EVAL_PERIOD = int(round(cfg.TEST.EVAL_PERIOD / scale)) |