Function to create a model that can run on many GPUs or CPUs. model_helper_obj: an object of ModelHelper input_builder_fun: Function that adds the input operators Note: Remember to instantiate reader outside of this
(
model_helper_obj,
input_builder_fun,
forward_pass_builder_fun,
param_update_builder_fun=None,
optimizer_builder_fun=None,
post_sync_builder_fun=None,
pre_grad_net_transformer_fun=None,
net_transformer_fun=None,
devices=None,
rendezvous=None,
net_type='dag',
broadcast_computed_params=True,
optimize_gradient_memory=False,
dynamic_memory_management=False,
blobs_to_keep=None,
use_nccl=False,
max_concurrent_distributed_ops=16,
cpu_device=False,
ideep=False,
num_threads_per_device=4,
shared_model=False,
combine_spatial_bn=False,
barrier_net_timeout_sec=_DEFAULT_BARRIER_NET_TIMEOUT_SEC,
)
| 47 | Parallelize(*args, **kwargs) |
| 48 | |
| 49 | def Parallelize( |
| 50 | model_helper_obj, |
| 51 | input_builder_fun, |
| 52 | forward_pass_builder_fun, |
| 53 | param_update_builder_fun=None, |
| 54 | optimizer_builder_fun=None, |
| 55 | post_sync_builder_fun=None, |
| 56 | pre_grad_net_transformer_fun=None, |
| 57 | net_transformer_fun=None, |
| 58 | devices=None, |
| 59 | rendezvous=None, |
| 60 | net_type='dag', |
| 61 | broadcast_computed_params=True, |
| 62 | optimize_gradient_memory=False, |
| 63 | dynamic_memory_management=False, |
| 64 | blobs_to_keep=None, |
| 65 | use_nccl=False, |
| 66 | max_concurrent_distributed_ops=16, |
| 67 | cpu_device=False, |
| 68 | ideep=False, |
| 69 | num_threads_per_device=4, |
| 70 | shared_model=False, |
| 71 | combine_spatial_bn=False, |
| 72 | barrier_net_timeout_sec=_DEFAULT_BARRIER_NET_TIMEOUT_SEC, |
| 73 | ): |
| 74 | ''' |
| 75 | Function to create a model that can run on many GPUs or CPUs. |
| 76 | model_helper_obj: an object of ModelHelper |
| 77 | input_builder_fun: |
| 78 | Function that adds the input operators |
| 79 | Note: Remember to instantiate reader outside of this |
| 80 | function so all devices share same reader object. |
| 81 | Signature: input_builder_fun(model) |
| 82 | forward_pass_builder_fun: |
| 83 | Function to add the operators to the model. |
| 84 | Must return list of loss-blob references that |
| 85 | are used to build the gradient. Loss scale parameter |
| 86 | is passed, as you should scale the loss of your model |
| 87 | by 1.0 / the total number of devices. |
| 88 | Signature: forward_pass_builder_fun(model, loss_scale) |
| 89 | param_update_builder_fun: |
| 90 | Function that adds operators that are run after |
| 91 | gradient update, such as updating the weights and |
| 92 | weight decaying. This is called for each GPU separately. |
| 93 | Signature: param_update_builder_fun(model) |
| 94 | optimizer_builder_fun: |
| 95 | Alternative to param_update_builder_fun, allows one |
| 96 | to add an optimizer for the whole model. Called only |
| 97 | once, without name or devicescope. |
| 98 | net_transformer_fun: |
| 99 | Optional function to transform the network after the |
| 100 | network is built. It will be called once (NOT once per |
| 101 | GPU.) |
| 102 | Signature: |
| 103 | net_transformer_fun( |
| 104 | model, num_devices, device_prefix, device_type) |
| 105 | pre_grad_net_transformer_fun: |
| 106 | Optional function to transform the network similar to |
no test coverage detected
searching dependent graphs…