A queue of works shared by all workers. A work queue is a queue that shares works for all workers. Any worker can use `take` or `input_producer` to take a work from this queue. On initialization, this queue will be populated by multiple epochs of work slices. Once failover happened, this qu
| 111 | # pylint: enable=unused-argument |
| 112 | |
| 113 | class WorkQueue(saver.BaseSaverBuilder.SaveableObject): |
| 114 | """A queue of works shared by all workers. |
| 115 | |
| 116 | A work queue is a queue that shares works for all workers. Any worker can use |
| 117 | `take` or `input_producer` to take a work from this queue. On initialization, |
| 118 | this queue will be populated by multiple epochs of work slices. Once failover |
| 119 | happened, this queue can be restored from latest checkpoint. |
| 120 | """ |
| 121 | class Resource(object): # pylint: disable=useless-object-inheritance |
| 122 | """Resource object of a work queue.""" |
| 123 | def __init__(self, name, works): |
| 124 | self._name = name |
| 125 | self._works = works |
| 126 | |
| 127 | @property |
| 128 | def name(self): |
| 129 | """Resource name of the work queue.""" |
| 130 | return self._name |
| 131 | |
| 132 | @property |
| 133 | def handle(self): |
| 134 | """Resource handle of the work queue.""" |
| 135 | return self._works._handle # pylint: disable=protected-access |
| 136 | |
| 137 | @property |
| 138 | def create(self): |
| 139 | """Resource creation op of the work queue.""" |
| 140 | return self._works._create # pylint: disable=protected-access |
| 141 | |
| 142 | @property |
| 143 | def is_initialized(self): |
| 144 | """Resource creation check op of the work queue.""" |
| 145 | return self._works._is_initialized # pylint: disable=protected-access |
| 146 | |
| 147 | def __init__( |
| 148 | self, |
| 149 | works, |
| 150 | num_epochs=1, |
| 151 | shuffle=True, |
| 152 | seed=None, |
| 153 | prefix=None, |
| 154 | num_slices=None, |
| 155 | num_clients=1, |
| 156 | name=None, |
| 157 | local_work_mgr=None): |
| 158 | """Constructs a work queue. |
| 159 | |
| 160 | Args: |
| 161 | works: A list of input paths. |
| 162 | num_epochs: (Optional.) An integer. If specified, this work queue |
| 163 | produces each work from `works` `num_epochs` times before |
| 164 | generating an `OutOfRange` error. 1 by default. |
| 165 | shuffle: (Optional.) Boolean. If true, the works are randomly shuffled |
| 166 | within each epoch. |
| 167 | seed: (Optional.) An integer. Seed used if shuffle == True. |
| 168 | prefix: (Optional.) Common prefix of all works. |
| 169 | num_slices: (Optional.) Total number of slices on all workers. |
| 170 | num_clients: (Optional.) Number of threads for taking works. |
no outgoing calls