3. Web APP
Task queue
Task queue
Task queue
WorkerWorkerWorker
DB
Push tasks
into queue
Workers pull task
from queue
Report progress
Pull DP for
report status
17. Gevent
gevent is a concurrency library based around libev. It
provides a clean API for a variety of concurrency and network
related tasks.
18. Greenlet
The primary pattern used in gevent is the Greenlet, a lightweight coroutine
provided to Python as a C extension module. Greenlets all run inside of
the OS process for the main program but are scheduled cooperatively.
Only one greenlet is ever running at any given time.
Spin-off of Stackless, a version of CPython that supports micro-threads
called “tasklets”. Tasklets run pseudo-concurrently (typically in a single or
a few OS-level threads) and are synchronized with data exchanges on
“channels”.
Its coroutine
19. The way to gevent
● Stackless
● Greenlet
● Eventlet
● Gevent
26. def echo(i):
time.sleep(0.001)
return i
# Non Deterministic Process Pool
from multiprocessing.pool import Pool
p = Pool(10)
run1 = [a for a in p.imap_unordered(echo, range(10))]
run2 = [a for a in p.imap_unordered(echo, range(10))]
run3 = [a for a in p.imap_unordered(echo, range(10))]
run4 = [a for a in p.imap_unordered(echo, range(10))]
print(run1 == run2 == run3 == run4)
False
Determinism
27. # Deterministic Gevent Pool
from gevent.pool import Pool
p = Pool(10)
run1 = [a for a in p.imap_unordered(echo, range(10))]
run2 = [a for a in p.imap_unordered(echo, range(10))]
run3 = [a for a in p.imap_unordered(echo, range(10))]
run4 = [a for a in p.imap_unordered(echo, range(10))]
print(run1 == run2 == run3 == run4)
True
Determinism
29. class MyGreenlet(Greenlet):
def __init__(self, message, n):
Greenlet.__init__(self)
self.message = message
self.n = n
def _run(self):
print(self.message)
gevent.sleep(self.n)
g = MyGreenlet("Hi there!", 3)
g.start()
g.join()
Spawning Greenlets
30. Greenlet State
started -- Boolean, indicates whether the Greenlet has been
started
ready() -- Boolean, indicates whether the Greenlet has halted
successful() -- Boolean, indicates whether the Greenlet has
halted and not thrown an exception
value -- arbitrary, the value returned by the Greenlet
exception -- exception, uncaught exception instance thrown
inside the greenlet
45. PEP 255 -- Simple Generators
Authors:
Neil Schemenauer <nas at arctrix.com>,
Tim Peters <tim.peters at gmail.com>,
Magnus Lie Hetland <magnus at hetland.org>
Created: 18-May-2001
Python 2.2
46. PEP 342 Coroutine via Enhanced Generators
Authors: Guido van Rossum, Phillip J. Eby
Created: 10-May-2005
Python 2.5
PEP 288, Generators Attributes and Exceptions
(Raymond Hettinger)
PEP 325, Resource-Release Support for Generators
(Samuele Pedroni)
47. What’s new?
1. (yield) statement:
value = (yield)
2. send()
3. throw()
4. close()
5. Ensure that close() is called when a generator iterator is
garbage-collected.
6. Allow yield to be used in try/finally blocks
50. RESULT = yield from EXPR
PEP 380 Syntax for Delegating to aSubgenerator
Authors: Gregory Ewing
Created: 13-Feb-2009
Python 3.3 - September 29, 2012
51. PEP 0492 -- Coroutines with async and await syntax
Authors: Yury Selivanov
Created: 09-Apr-2015
Python 3.5
53. Event loop
async def coroutine():
print('Running in coro')
event_loop = asyncio.get_event_loop()
try:
print('init coroutine')
coro = coroutine()
print('waiting for ...')
event_loop.run_until_complete(coro)
finally:
print('closing event loop')
event_loop.close()
init coroutine
waiting for ...
Running in coro
closing event loop
54. Coroutines, futures and tasks
● Coroutines
○ async def
○ @asyncio.coroutine
● Futures
○ promise of a result or an error
● Tasks
○ future which runs a coroutine
63. Summary
● В мире python много библиотек хороших и разных
● Некоторые библиотеки, когда-то популярные,
уступают свое место новым, более продуманным
● Чтобы крепче спать, не ковыряйтесь в потрохах
● Не всё то, чем кажется