Skip to content

Advanced Python 2 min read

TIL - Python 3.14 runs code in parallel with subinterpreters

InterpreterPoolExecutor gives you multi-core parallelism inside one process, each interpreter with its own GIL.

Python 3.14 added concurrent.futures.InterpreterPoolExecutor and the concurrent.interpreters module. Each subinterpreter has its own GIL, so CPU-bound Python code can run in parallel within a single process.

It’s a middle ground between threads and processes:

ThreadsSubinterpretersProcesses
CPU parallelism (standard build)shared GILone GIL each
Startup costtinysmalllarge
Shared Python objectsyesno (isolated)no
Isolationnonestrongfull

Using it looks exactly like the other executors:

from concurrent.futures import InterpreterPoolExecutor
import time

def cpu_work(n):
    return sum(i * i for i in range(n))

if __name__ == "__main__":
    start = time.perf_counter()
    with InterpreterPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(cpu_work, [2_000_000] * 4))
    print(len(results), f"{time.perf_counter() - start:.2f}s")
python

On a multi-core machine this scales like a process pool for pure-Python CPU work, but without spawning whole new processes.

The lower-level concurrent.interpreters module gives you direct control:

from concurrent import interpreters

interp = interpreters.create()
interp.exec("import sys; print('hello from', sys.implementation.name)")

print(interp.call(pow, 2, 10))   # 1024: run a function and get its result
interp.close()
python

interpreters.create_queue() passes data between interpreters.

What to know before using it:

  • Nothing is shared. Each interpreter imports modules separately and has its own globals. Arguments and results are copied between interpreters (usually via pickle), so the function has to be importable.
  • Extension module support varies. C extensions must opt in to multi-interpreter support, and many popular ones haven’t yet, so imports may fail inside a subinterpreter.
  • It’s new, and the ecosystem is still catching up. Treat it as promising rather than default.

Works on: Python 3.14+ (PEP 734).

Takeaway: for CPU-bound pure-Python work, try InterpreterPoolExecutor as a lighter alternative to ProcessPoolExecutor.