concurrent

Thread- and process-parallel versions for elements of your workflow.

class ProcessFork(call=(), *calls, max_workers=4, initializer=None, initargs=(), max_tasks_per_child=None, timeout=None)

Bases: IndentRepr, Generic[P, T]

Call multiple callables with the same argument(s) in parallel processes.

Generic type annotation of instances is recommended. Provide a list of one or more input types that all callables take, followed by a tuple specifying the concatenation of the return types of all callables, ignoring empty tuples. If only a single object remains, the type of that object should be annotated.

Parameters:

• call (callable or iterable of callables, optional) – One callable or an iterator of callables to all call with the same arguments. Defaults to an empty tuple.

• *calls (callable) – Additional callables to call with the same argument(s).

• max_workers (int, optional) – Maximum number of worker processes used in the pool to call calls asynchronously. Will be forwarded to the constructor of ProcessPoolExecutor. Defaults to 4.

• initializer (callable, optional) – Called at the start of each worker process. Will be forwarded to the constructor of ProcessPoolExecutor. Defaults to None.

• initargs (tuple, optional) – Arguments passed to the initializer. Will be forwarded to the constructor of ProcessPoolExecutor. Defaults to an empty tuple.

• max_tasks_per_child (int, optional) – Maximum number of iterable items to transform in each worker process before they are being restarted. Defaults to None, indicating no restart(s) at all.

• timeout (int or float, optional) – Maximum time (in seconds) to wait for results to be available. Defaults to None, which means there is no limit for the time to wait. Will be forwarded to the result method of Future.

See also

concurrent.futures.ProcessPoolExecutor

__call__(*args)

Concurrently call all specified calls with the same argument(s).

Parameters:

*args – Arguments to call all calls with.

Returns:

Concatenation of all return values of all calls in order. If only one of the calls returns something other than an empty tuple, that object is returned.

Return type:

tuple or object

Raises:

ForkError – When one of the calls raises an exception.

class ProcessMap(transform, wrapper=None, max_workers=4, initializer=None, initargs=(), max_tasks_per_child=None, timeout=None, chunksize=1)

Bases: ArgRepr, Generic[P, S, T]

Partial of concurrent.futures.ProcessPoolExecutor.map.

Upon subclassing and/or instantiation, type annotation with a list of the argument type(s) of transform, the return type of call, and the return type of wrapper is recommended.

Parameters:

• transform (callable) – Transforms element(s) of the input iterable(s).

• wrapper (type or callable, optional) – If not given, an attempt will be made to return the type of the first iterable the callable instance is being called with (by calling its class with a list of the mapped elements). If explicitly given, wrapper will be called with a list of mapped elements. Consequently, the return type will be the (return) type of wrapper.

• max_workers (int, optional) – Maximum number of worker processes used in the pool to execute transform asynchronously. Will be forwarded to the constructor of ProcessPoolExecutor. Defaults to 4.

• initializer (callable, optional) – Called at the start of each worker process. Will be forwarded to the constructor of ProcessPoolExecutor. Defaults to None.

• initargs (tuple, optional) – Arguments passed to the initializer. Will be forwarded to the constructor of ProcessPoolExecutor. Defaults to an empty tuple.

• max_tasks_per_child (int, optional) – Maximum number of iterable items to transform in each worker process before they are being restarted. Defaults to None indicating no restart(s) at all.

• timeout (int or float, optional) – Maximum time (in seconds) to wait for results to be available. Defaults to None, which means there is no limit for the time to wait. Will be forwarded to the map method of the ProcessPoolExecutor.

• chunksize (int, optional) – Number of items from the iterable to feed to one worker process at a time. Defaults to 1. Will be forwarded to the map method of the ProcessPoolExecutor.

Note

In contrast to calling the map method of a ProcessPoolExecutor directly, which returns a generator object, the mapped iterable is fully manifested first and only then wrapped.

See also

concurrent.futures.ProcessPoolExecutor

__call__(iterable, *iterables)

Concurrently transform the element(s) of the given iterable(s).

Parameters:

• iterable (Iterable) – An iterable of elements to transform.

• *iterables (Iterable) – If given, the cached transform is called with the corresponding elements of ìterable and all iterables as arguments.

Returns:

Same type as iterable if wrapper was not specified on instantiation or the (return) type of wrapper. Note that, as with python’s builtin map function, the length of the output sequence is limited by the shortest of the input iterables.

Return type:

Sequence

Raises:

MapError – If calling the ProcessPoolExecutor’s map method raises an exception or if wrapping the results leads to an exception.

class ThreadFork(call=(), *calls, max_workers=16, thread_name_prefix='', initializer=None, initargs=(), timeout=None)

Bases: IndentRepr, Generic[P, T]

Call multiple callables with the same argument(s) in parallel threads.

Generic type annotation of instances is recommended. Provide a list of one or more input types that all callables take, followed by a tuple specifying the concatenation of the return types of all callables, ignoring empty tuples. If only a single object remains, the type of that object should be annotated.

Parameters:

• call (callable or iterable of callables, optional) – One callable or an iterator of callables to all call with the same arguments. Defaults to an empty tuple.

• *calls (callable) – Additional callables to call with the same argument(s).

• max_workers (int, optional) – Maximum number of worker threads used in the pool to call calls asynchronously. Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to 16.

• thread_name_prefix (str, optional) – Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to an empty string.

• initializer (callable, optional) – Called at the start of each worker thread. Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to None.

• initargs (tuple, optional) – Arguments passed to the initializer. Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to an empty tuple.

• timeout (int or float, optional) – Maximum time (in seconds) to wait for results to be available. Defaults to None, which means there is no limit for the time to wait. Will be forwarded to the result method of Future.

See also

concurrent.futures.ThreadPoolExecutor

__call__(*args)

Concurrently call all specified calls with the same argument(s).

Parameters:

*args – Arguments to call all calls with.

Returns:

Concatenation of all return values of all calls in order. If only one of the calls returns something other than an empty tuple, that object is returned.

Return type:

tuple or object

Raises:

ForkError – When one of the calls raises an exception.

class ThreadMap(transform, wrapper=None, max_workers=16, thread_name_prefix='', initializer=None, initargs=(), timeout=None)

Bases: ArgRepr, Generic[P, S, T]

Partial of concurrent.futures.ThreadPoolExecutor.map.

Upon subclassing and/or instantiation, type annotation with a list of the argument type(s) of transform, the return type of call, and the return type of wrapper is recommended.

Parameters:

• transform (callable) – Transforms element(s) of the input iterable(s).

• wrapper (type or callable, optional) – If not given, an attempt will be made to return the type of the first iterable the callable instance is being called with (by calling its class with a list of the mapped elements). If explicitly given, wrapper will be called with a list of mapped elements. Consequently, the return type will be the (return) type of wrapper.

• max_workers (int, optional) – Maximum number of worker threads used in the pool to execute transform asynchronously. Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to 16.

• thread_name_prefix (str, optional) – Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to an empty string.

• initializer (callable, optional) – Called at the start of each worker thread. Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to None.

• initargs (tuple, optional) – Arguments passed to the initializer. Will be forwarded to the constructor of ThreadPoolExecutor. Defaults to an empty tuple.

• timeout (int or float, optional) – Maximum time (in seconds) to wait for results to be available. Defaults to None, which means there is no limit for the time to wait. Will be forwarded to the map method of the ThreadPoolExecutor.

Note

In contrast to calling the map method of a ThreadPoolExecutor directly, which returns a generator object, the mapped iterable is fully manifested first and only then wrapped.

See also

concurrent.futures.ThreadPoolExecutor

__call__(iterable, *iterables)

Concurrently transform the element(s) of the given iterable(s).

Parameters:

• iterable (Iterable) – An iterable of elements to transform.

• *iterables (Iterable) – If given, the cached transform is called with the corresponding elements of ìterable and all iterables as arguments.

Returns:

Same type as iterable if wrapper was not specified on instantiation or the (return) type of wrapper. Note that, as with python’s builtin map function, the length of the output sequence is limited by the shortest of the input iterables.

Return type:

Sequence

Raises:

MapError – If calling the ThreadPoolExecutor’s map method raises an exception or if wrapping the results leads to an exception.