Returns list of results from `pagefn()`, optionally using concurrency for speed. Args: path: Path of document. pagefn: Function to call for each page; is passed (page, *pagefn_args, **pagefn_kwargs). Return value is added to list
(
path,
pagefn,
*,
pagefn_args=(),
pagefn_kwargs=dict(),
initfn=None,
initfn_args=(),
initfn_kwargs=dict(),
pages=None,
method='single',
concurrency=None,
_stats=False,
)
| 24646 | |
| 24647 | |
| 24648 | def apply_pages( |
| 24649 | path, |
| 24650 | pagefn, |
| 24651 | *, |
| 24652 | pagefn_args=(), |
| 24653 | pagefn_kwargs=dict(), |
| 24654 | initfn=None, |
| 24655 | initfn_args=(), |
| 24656 | initfn_kwargs=dict(), |
| 24657 | pages=None, |
| 24658 | method='single', |
| 24659 | concurrency=None, |
| 24660 | _stats=False, |
| 24661 | ): |
| 24662 | ''' |
| 24663 | Returns list of results from `pagefn()`, optionally using concurrency for |
| 24664 | speed. |
| 24665 | |
| 24666 | Args: |
| 24667 | path: |
| 24668 | Path of document. |
| 24669 | pagefn: |
| 24670 | Function to call for each page; is passed (page, *pagefn_args, |
| 24671 | **pagefn_kwargs). Return value is added to list that we return. If |
| 24672 | `method` is not 'single', must be a top-level function - nested |
| 24673 | functions don't work with concurrency. |
| 24674 | pagefn_args |
| 24675 | pagefn_kwargs: |
| 24676 | Additional args to pass to `pagefn`. Must be picklable. |
| 24677 | initfn: |
| 24678 | If true, called once in each worker process; is passed |
| 24679 | (*initfn_args, **initfn_kwargs). |
| 24680 | initfn_args |
| 24681 | initfn_kwargs: |
| 24682 | Args to pass to initfn. Must be picklable. |
| 24683 | pages: |
| 24684 | List of page numbers to process, or None to include all pages. |
| 24685 | method: |
| 24686 | 'single' |
| 24687 | Do not use concurrency. |
| 24688 | 'mp' |
| 24689 | Operate concurrently using Python's `multiprocessing` module. |
| 24690 | 'fork' |
| 24691 | Operate concurrently using custom implementation with |
| 24692 | `os.fork()`. Does not work on Windows. |
| 24693 | concurrency: |
| 24694 | Number of worker processes to use when operating concurrently. If |
| 24695 | None, we use the number of available CPUs. |
| 24696 | _stats: |
| 24697 | Internal, may change or be removed. If true, we output simple |
| 24698 | timing diagnostics. |
| 24699 | |
| 24700 | Note: We require a file path rather than a Document, because Document |
| 24701 | instances do not work properly after a fork - internal file descriptor |
| 24702 | offsets are shared between the parent and child processes. |
| 24703 | ''' |
| 24704 | if _stats: |
| 24705 | t0 = time.time() |