diff --git a/peps/pep-0828.rst b/peps/pep-0828.rst index 55fa1ba0b1d..54b9078d633 100644 --- a/peps/pep-0828.rst +++ b/peps/pep-0828.rst @@ -28,9 +28,8 @@ For example, the following code is valid under this PEP: yield from generator() - -In addition, this PEP introduces a new ``async yield from`` syntax to use -existing ``yield from`` semantics on an asynchronous generator: +In addition, this PEP introduces a new ``async yield from`` construct to +delegate to an asynchronous generator: .. code-block:: python @@ -67,6 +66,10 @@ as an :term:`asynchronous generator`, sometimes suffixed with "function". In contrast, the object returned by an asynchronous generator is referred to as an :term:`asynchronous generator iterator` in this PEP. +This PEP also uses the term "subgenerator" to refer to a generator, synchronous +or asynchronous, that is used inside of a ``yield from`` or ``async yield from`` +expression. + Motivation ========== @@ -105,8 +108,8 @@ in asynchronous generators: 2. https://discuss.python.org/t/47050 3. https://discuss.python.org/t/66886 -Additionally, this design decision has `come up -`__ on Stack Overflow. +Additionally, users have `questioned `__ +this design decision on Stack Overflow. Subgenerator delegation is useful for asynchronous generators @@ -119,16 +122,17 @@ item. This comes with a few drawbacks: 1. It obscures the intent of the code and increases the amount of effort necessary to work with asynchronous generators, because each delegation point becomes a loop. This damages the power of asynchronous generators. -2. :meth:`~agen.asend`, :meth:`~agen.athrow`, and :meth:`~agen.aclose`, +2. :meth:`~agen.asend`, :meth:`~agen.athrow`, and :meth:`~agen.aclose` do not interact properly with the caller. This is the primary reason that ``yield from`` was added in the first place. 3. Return values are not natively supported with asynchronous generators. The - workaround for this it to raise an exception, which increases boilerplate. + workaround for this is to raise an exception, which increases boilerplate. Specification ============= + Syntax ------ @@ -163,7 +167,7 @@ This PEP retains all existing ``yield from`` semantics; the only detail is that asynchronous generators may now use it. Because the existing ``yield from`` behavior may only yield from a synchronous -generator, this is true for asynchronous generators as well. +subgenerator, this is true for asynchronous generators as well. For example: @@ -294,6 +298,7 @@ knowledge of ``yield from`` in synchronous generators. Potential footguns ------------------ + Forgetting to ``await`` a future ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -314,18 +319,45 @@ For example: await asyncio.sleep(0.25) return [1, 2, 3] - async def generator(): + async def agenerator(): # Forgot to await! yield from asyncio.ensure_future(steps()) async def run(): total = 0 - async for i in generator(): + async for i in agenerator(): # TypeError?! total += i print(total) +Attempting to use ``yield from`` on an asynchronous subgenerator +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A common intuition among developers is that ``yield from`` inside an +asynchronous generator will also delegate to another asynchronous generator. +As such, many users were surprised to see that, in this proposal, the following +code is invalid: + +.. code-block:: python + + async def asubgenerator(): + yield 1 + yield 2 + + async def agenerator(): + yield from asubgenerator() + + +As a solution, when ``yield from`` is given an object that is not iterable, +the implementation can detect if that object is asynchronously iterable. +If it is, ``async yield from`` can be suggested in the exception message. + +This is done in the reference implementation of this proposal; the example +above raises a :exc:`TypeError` that reads ``async_generator object is not +iterable. Did you mean 'async yield from'?`` + + Reference Implementation ======================== @@ -336,7 +368,77 @@ A reference implementation of this PEP can be found at Rejected Ideas ============== -TBD. + +Using ``yield from`` to delegate to asynchronous generators +----------------------------------------------------------- + +It has been argued that many developers may intuitively believe that using a +plain ``yield from`` inside an asynchronous generator would also delegate to +an asynchronous subgenerator rather than a synchronous subgenerator. As such, +it was proposed to make ``yield from`` always delegate to an asynchronous +subgenerator. + +For example: + +.. code-block:: python + + async def asubgenerator(): + yield 1 + yield 2 + + async def agenerator(): + yield from asubgenerator() + + +This was rejected, primarily because it felt very wrong for ``yield from x`` to +be valid or invalid depending on the type of generator it was used in. + +In addition, there is no precedent for this kind of behavior in Python; inherently +synchronous constructs always have an asynchronous counterpart for use in +asynchronous functions, instead of implicitly switching protocols depending on +the type of function it is used in. For example, :keyword:`with` always means that the +:term:`synchronous context management protocol ` will +be invoked, even when used in an ``async def`` function. + +Finally, this would leave a gap in asynchronous generators, because there would be +no mechanism for delegating to a synchronous subgenerator. Even if this is not a +common pattern today, this may become common in the future, in which case it would +be very difficult to change the meaning of ``yield from`` in an asynchronous +generator. + + +Letting ``yield from`` determine which protocol to use +------------------------------------------------------ + +As a solution to the above rejected idea, it was proposed to allow ``yield from x`` +to invoke the synchronous or asynchronous generator protocol depending on the type +of ``x``. In turn, this would allow developers to delegate to both synchronous +and asynchronous subgenerators while continuing to use the familiar ``yield from`` +syntax. + +For example: + +.. code-block:: python + + async def asubgenerator(): + yield 1 + yield 2 + + async def agenerator(): + yield from asubgenerator() + yield from range(3, 5) + + +Mechanically, this is possible, but the exact behavior will likely be counterintuitive +and ambigious. In particular: + +1. If an object implements both :meth:`~object.__iter__` and :meth:`~object.__aiter__`, + it's not clear which protocol Python should choose. +2. If the chosen protocol raises an exception, should the exception be propagated, or + should Python try to use the other protocol first? + +Additionally, this approach is inherently slower, because of the additional overhead +of detecting which generator protocol to use. Acknowledgements @@ -344,9 +446,9 @@ Acknowledgements Thanks to Bartosz Sławecki for aiding in the development of the reference implementation of this PEP. In addition, the :exc:`StopAsyncIteration` -changes in addition to the support for non-``None`` return values inside +changes alongside the support for non-``None`` return values inside asynchronous generators were largely based on Alex Dixon's design from -`python/cpython#125401 `__ +`python/cpython#125401 `__. Change History