I’m working in a project with Python and AIOHTTP, so inevitably I was going to hit some threading issues that forced me to learn more about asyncio. I’ve collected in this blogpost some useful links. Please contact me for any other good stuff that you can suggest :)
- Official docs:
- Coroutines and tasks.
- Executing code in thread of process pools: contains three snippets for the three main alternatives: running in the default loop’s executor, in a custom thread pool and a custom process pool.
- PEP 525.
- Async Python: The Different Forms of Concurrency: very good summary about the current state of Python parallelism, including GIL explanation and a “Making the Right Choice” recommendation.
- How to deal with blocking code within asyncio event loop: recommendations, including lower-level suggestions than usual.
- Async IO in Python: A Complete Walkthrough.
- Explanation about
run_in_executorusingThreadPoolExecutor, which is affected by the GIL. - Yet another async/await tutorial.
- uvloop, an alternative implementation of the built-in asyncio event loop.
- asyncio snippets.
- To explore: Python 3.8 multiprocessing shared memory: Python 3.8, currently in beta, will have a
multiprocessing.shared_memorymodule. That might make feasible sharing the connection among processes. I haven’t done any research about this, so this can be plain wrong. In addition, the overhead of creating a new process might make this approach prohibitive.
Extra balls
Fast.ai
Fast.ai is a library for neural networks that has a useful helper for triggering processes (via @antor, thanks!):
from fastai.core import parallel
def process_one(item,i):
return (item**2, i)
result = parallel(process_one, range(100000))
Talks
- Dynamic languages and Parallelism: How to Go from Broken or Slow to Safe and Efficient: not about Python but dynamic languages in general.