MCPcopy Create free account
hub / github.com/cloudpipe/cloudpickle / _find_imported_submodules

Function _find_imported_submodules

cloudpickle/cloudpickle.py:340–386  ·  view source on GitHub ↗

Find currently imported submodules used by a function. Submodules used by a function need to be detected and referenced for the function to work correctly at depickling time. Because submodules can be referenced as attribute of their parent package (``package.submodule``), we need a

(code, top_level_dependencies)

Source from the content-addressed store, hash-verified

338
339
340def _find_imported_submodules(code, top_level_dependencies):
341 """Find currently imported submodules used by a function.
342
343 Submodules used by a function need to be detected and referenced for the
344 function to work correctly at depickling time. Because submodules can be
345 referenced as attribute of their parent package (``package.submodule``), we
346 need a special introspection technique that does not rely on GLOBAL-related
347 opcodes to find references of them in a code object.
348
349 Example:
350 ```
351 import concurrent.futures
352 import cloudpickle
353 def func():
354 x = concurrent.futures.ThreadPoolExecutor
355 if __name__ == '__main__':
356 cloudpickle.dumps(func)
357 ```
358 The globals extracted by cloudpickle in the function's state include the
359 concurrent package, but not its submodule (here, concurrent.futures), which
360 is the module used by func. Find_imported_submodules will detect the usage
361 of concurrent.futures. Saving this module alongside with func will ensure
362 that calling func once depickled does not fail due to concurrent.futures
363 not being imported
364 """
365
366 subimports = []
367 # check if any known dependency is an imported package
368 for x in top_level_dependencies:
369 if (
370 isinstance(x, types.ModuleType)
371 and hasattr(x, "__package__")
372 and x.__package__
373 ):
374 # check if the package has any currently loaded sub-imports
375 prefix = x.__name__ + "."
376 # A concurrent thread could mutate sys.modules,
377 # make sure we iterate over a copy to avoid exceptions
378 for name in list(sys.modules):
379 # Older versions of pytest will add a "None" module to
380 # sys.modules.
381 if name is not None and name.startswith(prefix):
382 # check whether the function can address the sub-module
383 tokens = set(name[len(prefix) :].split("."))
384 if not tokens - set(code.co_names):
385 subimports.append(sys.modules[name])
386 return subimports
387
388
389# relevant opcodes

Callers 1

_function_getstateFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected