Inconsistent `.extend()` behavior in bytearray

[tim-one]

Bug report

Bug description:

Extending a bytearrary can behave very differently when extending via an iterable than when extending a list or array.array in the same way:

>>> from itertools import islice
>>> xs = [0, 1, 2]
>>> xs.extend(islice(xs, 12000))
>>> len(xs)
12003
>>> import array
>>> xs = array.array('Q', [0, 1, 2])
>>> xs.extend(islice(xs, 12000))
>>> len(xs)
12003

Tricky, but it's always worked this way, and is very convenient to extend a sequence with copies of itself. Note that the iterator picks up new elements of the sequence while they're being added.

bytearray doesn't work this way, though. It appears to capture the sequence's length just once at the start, and so can't do more than double the original length.

>>> xs = bytearray([0, 1, 2])
>>> xs.extend(islice(xs, 12000))
>>> len(xs)
6
>>> list(xs)
[0, 1, 2, 0, 1, 2]

Bumped into this when changing old code to switch from lists of small ints to bytearrays instead. Quite a head-scratcher to figure out what went wrong! ;-)

CPython versions tested on:

3.15, 3.14

Operating systems tested on:

No response

[cmaloney]

That bytearray appears to capture the sequence's length just once at the start is right on the nose. Going through the implementation bytearray.extend doesn't extend the underlying bytearray until the iterator is exhausted. bytearray.extend when passed an iterable that doesn't support the buffer protocol starts by allocating a temporary bytearray then exhausts the iterator into that temporary resizing as needed. Only once it has exhausted the iterator does it extend the current object with the newly constructed temporary.

List on the other hand (list_extend_iter_lock_held) continually resizes itself while doing the extend. In terms of which is the "right" behavior sequence.extend suggests the length 6 is the general sequence behavior pattern:

>>> xs = [1,2,3]
>>> xs[len(xs):len(xs)] = xs
>>> xs
[1, 2, 3, 1, 2, 3]

The behavior has not changed for a long time in bytearray (since at least 2009 in my look at git blame), and the behavior of list looks to be of a similar vintage so I suspect this is better to document than to normalize. Code which assumes the behavior almost certainly exists and while standardizing would be nice it probably doesn't pass the PEP-387 test of "a large benefit to breakage ratio".

Secondary bug to me: itertools.islice doesn't support __length_hint__ which means things are less efficient then they could be in the .extend

[tim-one]

Ya, it's possible (likely!) nothing can be changed here. Lists have always defined what happens if the length changes during iteration, and list.extend() follows that model. array.array followed suit. bytearray is the oddball. `coillections.deque' does a third thing:

RuntimeError: deque mutated during iteration

The docs:

For the most part, this is the same as writing seq[len(seq):len(seq)] = **iterable.**

isn't of much help, as it doesn't give a clue about what "the most part" means 😦