-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
PEP 841: Adding Frozen Syntax to Make Immutable Types Optimizable #5049
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
305f540
96feb52
5d6d9cb
37856a6
4708f2e
48ae072
3506d9b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,314 @@ | ||||||||||
| PEP: 841 | ||||||||||
| Title: Adding Frozen Syntax to Make Immutable Types Optimizable | ||||||||||
| Author: Donghee Na <donghee.na@python.org>, | ||||||||||
| Nikita Sobolev <mail@sobolevn.me> | ||||||||||
| Discussions-To: https://discuss.python.org/t/pep-841-adding-frozen-syntax-to-make-immutable-types-optimizable/108219 | ||||||||||
| Status: Draft | ||||||||||
| Type: Standards Track | ||||||||||
| Created: 16-Jul-2026 | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Day the PEP number was assigned:
Suggested change
|
||||||||||
| Python-Version: 3.16 | ||||||||||
| Post-History: `17-Jul-2026 <https://discuss.python.org/t/pep-841-adding-frozen-syntax-to-make-immutable-types-optimizable/108219>`__ | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
|
|
||||||||||
| Abstract | ||||||||||
| ======== | ||||||||||
|
|
||||||||||
| This PEP proposes *frozen display* syntax: ``f{1, 2, 3}`` evaluates to a | ||||||||||
| :class:`frozenset`, and ``f{'a': 1}`` evaluates to a ``frozendict``. | ||||||||||
| Because immutability is guaranteed by the syntax itself rather than | ||||||||||
| inferred from usage, the compiler can treat frozen displays as first-class | ||||||||||
| citizens of its optimization pipeline: constant displays are folded into a | ||||||||||
| single ``LOAD_CONST`` with an exact result type at compile time and | ||||||||||
| cached in ``.pyc`` files. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Motivation | ||||||||||
| ========== | ||||||||||
|
|
||||||||||
| Python has display syntax for its mutable containers but none for its | ||||||||||
| immutable ones. Today an immutable set must be written as | ||||||||||
| ``frozenset({1, 2, 3})`` and an immutable mapping as | ||||||||||
| ``frozendict({'a': 1})``. Each of these: | ||||||||||
|
|
||||||||||
| * builds a mutable set or dict, then copies it into the immutable type. | ||||||||||
| * looks up the name ``frozenset`` or ``frozendict`` at runtime on every | ||||||||||
| execution. | ||||||||||
| * cannot be easily optimized by the compiler, because either name may be | ||||||||||
| rebound and the call may have arbitrary side effects, or | ||||||||||
| value can be of unexpected type for the optimization, | ||||||||||
| value can be non uniquely referenced. | ||||||||||
|
|
||||||||||
| CPython already hints at the opportunity: the peephole optimizer rewrites | ||||||||||
| a constant set display into a frozenset, but only as the right operand | ||||||||||
| of ``in``. Assign the same display to a variable and the optimization is | ||||||||||
| gone. The root cause is that the compiler can never prove immutability | ||||||||||
| of a ``set`` or ``dict`` display, so it must rebuild it on every | ||||||||||
| execution. A display whose *semantics* guarantee immutability removes | ||||||||||
| that barrier once and for all. | ||||||||||
|
|
||||||||||
| One of the goals of this PEP is to increase the use of immutable containers | ||||||||||
| in CPython, preparing for a concurrent future in which free-threading and | ||||||||||
| subinterpreters become significantly more common. | ||||||||||
| Efficient creation of immutable data structures and convenient syntax would | ||||||||||
| encourage the use of ``frozendict`` and ``frozenset``. This would reduce bugs | ||||||||||
| caused by accidental mutation of shared mutable containers while also | ||||||||||
| improving performance. For example, subinterpreters already share | ||||||||||
| ``frozenset`` objects, and there are plans to share ``frozendict`` objects as | ||||||||||
| well. | ||||||||||
|
|
||||||||||
| Immutable container displays have been requested and discussed by the | ||||||||||
| community several times, most recently in the `frozenset and frozendict | ||||||||||
| comprehensions | ||||||||||
| <https://discuss.python.org/t/frozenset-and-frozendict-comprehensions/101584>`__ | ||||||||||
| thread on Discourse. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Rationale | ||||||||||
| ========= | ||||||||||
|
|
||||||||||
| Syntax, not a builtin call | ||||||||||
| -------------------------- | ||||||||||
|
|
||||||||||
| Only syntax gives the compiler a semantic guarantee. A call to | ||||||||||
| ``frozenset(...)`` can be shadowed, but a frozen display cannot. Every | ||||||||||
| optimization described below follows from this single property. | ||||||||||
|
|
||||||||||
| Static analysis benefits in the same way. Today, tools | ||||||||||
| that don't perform semantic analysis must assume | ||||||||||
| that ``frozenset(...)`` refers to the builtin. A frozen display turns | ||||||||||
| that assumption into a syntactic guarantee, so analyzers can treat the | ||||||||||
| result as immutable with full confidence. This holds even for purely | ||||||||||
| syntactic tools that perform no name resolution. | ||||||||||
|
|
||||||||||
| Linters and formatters can automatically rewrite ``frozenset({1, 2, 3})`` | ||||||||||
| and ``frozendict({1: 2})`` to be ``f{1, 2, 3}`` and ``f{1: 2}`` | ||||||||||
| on newer Python versions. | ||||||||||
|
|
||||||||||
| Why ``f{...}`` | ||||||||||
| -------------- | ||||||||||
|
|
||||||||||
| The ``f`` prefix reads as *frozen*, mirroring the familiar f-string | ||||||||||
| prefix convention. Sharing the letter with f-strings is not a problem: | ||||||||||
| strings are immutable too, so either way an ``f`` prefixed expression | ||||||||||
| evaluates to an immutable value. ``f{`` is a syntax error in all | ||||||||||
| current Python versions, so the syntax is fully backward compatible. The tokenizer | ||||||||||
| emits a single ``FLBRACE`` token for ``f{``, so ``f {1}`` (with a space) | ||||||||||
| remains an error and there is no ambiguity with the name ``f`` or with | ||||||||||
| f-strings. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Specification | ||||||||||
| ============= | ||||||||||
|
|
||||||||||
| Grammar | ||||||||||
| ------- | ||||||||||
|
|
||||||||||
| Our goal is to make new syntax and grammar identical | ||||||||||
| to existing ``set`` and ``dict`` syntax and grammar rules. | ||||||||||
|
|
||||||||||
| New alternatives are added to ``atom``, mirroring ``set`` and ``dict`` | ||||||||||
| displays and comprehensions:: | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| fset: FLBRACE star_named_expressions '}' | ||||||||||
| fsetcomp: FLBRACE star_named_expression for_if_clauses '}' | ||||||||||
| fdict: FLBRACE [double_starred_kvpairs] '}' | ||||||||||
| fdictcomp: FLBRACE kvpair for_if_clauses '}' | ||||||||||
|
|
||||||||||
| * ``f{1, 2, 3}`` is a frozenset display. | ||||||||||
| * ``f{'a': 1, 'b': 2}`` is a frozendict display. | ||||||||||
| * ``f{}`` is an empty frozendict, mirroring ``{}``. | ||||||||||
| * Star unpacking follows the existing displays: ``f{*xs}`` is a | ||||||||||
| frozenset display (like ``{*xs}``) and ``f{**d}`` is a frozendict | ||||||||||
| display (like ``{**d}``). | ||||||||||
| * Comprehensions are supported: ``f{x for x in xs}`` is a frozenset | ||||||||||
| comprehension and ``f{k: v for k, v in items}`` is a frozendict | ||||||||||
| comprehension. | ||||||||||
| * Async comprehensions are also supported | ||||||||||
| in async contexts: ``f{x async for arange(5)}`` | ||||||||||
| is a frozenset async comprehension | ||||||||||
| and ``f{k: v assync for k, v in items}`` | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
| is a frozendict async comprehension. | ||||||||||
| * :pep:`798` and unpacking in comprehensions is also supported. | ||||||||||
| ``f{*nums for nums in list_of_nums}`` is a frozenset | ||||||||||
| compehension with unpacking and ``f{**items for nums in list_of_items}`` | ||||||||||
| is a frozendict comprehension with unpacking. | ||||||||||
|
|
||||||||||
| AST | ||||||||||
| --- | ||||||||||
|
|
||||||||||
| Four new expression nodes are added: ``FrozenSet(elts)``, | ||||||||||
| ``FrozenDict(keys, values)``, ``FrozenSetComp(elt, generators)``, and | ||||||||||
| ``FrozenDictComp(key, value, generators)``, structurally identical to | ||||||||||
| their mutable counterparts. Distinct nodes (rather than a flag) let | ||||||||||
| every downstream | ||||||||||
| consumer (e.g. the symbol table, the AST optimizer, the code generator, | ||||||||||
| and third-party tools) dispatch on immutability directly. | ||||||||||
|
|
||||||||||
| Semantics | ||||||||||
| --------- | ||||||||||
|
|
||||||||||
| A frozenset display evaluates to exactly what ``frozenset({...})`` | ||||||||||
| returns. A frozendict display evaluates to exactly what | ||||||||||
| ``frozendict({...})`` returns. Both result types are immutable and | ||||||||||
| hashable, which is what makes the compile-time treatment below sound. | ||||||||||
|
|
||||||||||
| Bytecode | ||||||||||
| -------- | ||||||||||
|
|
||||||||||
| Two new instructions are added: | ||||||||||
|
|
||||||||||
| * ``BUILD_FROZENSET (count)`` works like ``BUILD_SET``, but the freshly | ||||||||||
| created, uniquely referenced set is frozen in place with no copy. | ||||||||||
| * ``BUILD_FROZENMAP (count)`` works like ``BUILD_MAP``, but creates a | ||||||||||
| ``frozendict``. | ||||||||||
|
|
||||||||||
| ``count`` is an ordinary oparg with the same format and meaning as in | ||||||||||
| the existing ``BUILD_SET`` and ``BUILD_MAP`` instructions. | ||||||||||
|
|
||||||||||
| Displays that use star-unpacking or exceed the stack-use guideline fall | ||||||||||
| back to building the mutable container and freezing it in place. The | ||||||||||
| result is indistinguishable. Comprehensions take the same path, so | ||||||||||
| they need no new opcodes. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| The optimization pipeline | ||||||||||
| ========================= | ||||||||||
|
|
||||||||||
| The central claim of this PEP is that frozen displays are not merely | ||||||||||
| convenient syntax: they give every stage of the compiler a guarantee it | ||||||||||
| can act on. The reference implementation already exercises the full | ||||||||||
| pipeline: | ||||||||||
|
|
||||||||||
| 1. **AST preprocessing.** ``FrozenSet`` and ``FrozenDict`` participate | ||||||||||
| in AST-level constant folding of their elements. | ||||||||||
|
|
||||||||||
| 2. **Code generation.** The common case compiles to a single | ||||||||||
| ``BUILD_FROZENSET`` / ``BUILD_FROZENDICT`` instruction with no name | ||||||||||
| lookup, no temporary copy, and an exact, statically known result type | ||||||||||
| (used by the compiler's type inference, e.g. to reject | ||||||||||
| ``f{1, 2}[0]`` at compile time). | ||||||||||
|
|
||||||||||
| 3. **Control flow graph (CFG) constant folding.** A display whose | ||||||||||
| keys and values are all constants is folded into a single | ||||||||||
| ``LOAD_CONST``, serialized into the ``.pyc`` by marshal, and shared | ||||||||||
| across all executions: zero per-execution construction cost. Unlike | ||||||||||
| the existing list/set folds, this is *unconditionally* valid, since | ||||||||||
| immutability comes from the language semantics, not from how the | ||||||||||
| value is used. A display with a non-constant element, e.g. | ||||||||||
| ``f{'key': ['list']}``, is still built at runtime. | ||||||||||
|
|
||||||||||
| 4. **Constant deduplication.** Frozen constants participate in | ||||||||||
| ``co_consts`` deduplication. For a frozendict the deduplication key | ||||||||||
| keeps the insertion order, so a display never loses its iteration | ||||||||||
| order to an equal display with a different key order. | ||||||||||
|
|
||||||||||
| .. note:: | ||||||||||
|
|
||||||||||
| This PEP deliberately makes no claims about JIT-level optimization: | ||||||||||
| the JIT project is currently on hold following the `Steering | ||||||||||
| Council's announcement | ||||||||||
| <https://discuss.python.org/t/an-announcement-from-the-steering-council-regarding-the-jit-project/107638>`__. | ||||||||||
|
|
||||||||||
| The pipeline also opens future work that mutable displays can never | ||||||||||
| support: sharing folded frozen constants across code objects and | ||||||||||
| immortalizing them under free threading. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Backwards Compatibility | ||||||||||
| ======================= | ||||||||||
|
|
||||||||||
| ``f{`` is a syntax error today, so no existing code changes meaning. | ||||||||||
| The changes visible to tooling are: a new ``FLBRACE`` token, four new | ||||||||||
| AST node types, two new opcodes, and a bytecode magic number bump. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| How to Teach This | ||||||||||
| ================= | ||||||||||
|
|
||||||||||
| "Prefix a set or dict display with ``f`` to make it frozen". | ||||||||||
| Style guidance: prefer ``f{...}`` over | ||||||||||
| ``frozenset({...})`` for literal values on Python 3.16+. | ||||||||||
| Constant frozen displays are free after the first execution. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Impact on the Standard Library | ||||||||||
| ============================== | ||||||||||
|
|
||||||||||
| A quick survey of the standard library (excluding tests) finds about | ||||||||||
| 105 ``frozenset(...)`` and 65 ``frozendict(...)`` call sites, of which | ||||||||||
| about 46 and 22 respectively pass a literal display and could be | ||||||||||
| written as ``f{...}``. They spread across widely used modules such as | ||||||||||
| ``typing``, ``dataclasses``, ``functools``, ``copy``, and | ||||||||||
| ``traceback``. | ||||||||||
|
|
||||||||||
| These numbers are only an estimate of the potential effect. This PEP | ||||||||||
| does not propose a mechanical rewrite of the standard library. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Reference Implementation | ||||||||||
| ======================== | ||||||||||
|
|
||||||||||
| A complete implementation, including the parser, AST, code generator, | ||||||||||
| and CFG constant folding, is available in the `fset_fdict branch | ||||||||||
| <https://github.com/corona10/cpython/tree/fset_fdict>`__ of the author's | ||||||||||
| CPython fork. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Rejected Ideas | ||||||||||
| ============== | ||||||||||
|
|
||||||||||
| Alternative spellings | ||||||||||
| --------------------- | ||||||||||
|
|
||||||||||
| Many spellings were considered. ``f`` was chosen simply because it is | ||||||||||
| the prefix that best evokes *frozen*: | ||||||||||
|
|
||||||||||
| * Single letter prefixes: ``i{'key': 1}`` (immutable), ``z{'key': 1}``. | ||||||||||
| This was rejected because types are called ``frozen``, | ||||||||||
| ``f`` as a prefix reads the best. | ||||||||||
| * Multi letter prefixes: ``fr{'key': 1}``, ``fz{'key': 1}``, | ||||||||||
| ``frz{'key': 1}``. This was rejected because it just adds an extra letter | ||||||||||
| to write and read with no extra real value. | ||||||||||
| * Symbol prefixes: ``${'key': 1}``, ``+{'key': 1}``. | ||||||||||
| This was rejected because most symbols already have a meaning as operators. | ||||||||||
| We can't reuse them for this new purpose. | ||||||||||
| We also don't want to add new symbols like ``$`` | ||||||||||
| to keep them for something else in the future. | ||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I must confess I'm slightly surprised to see direct mention here about the risk of monopolizing symbols such as In particular since this PEP is all about adding constant folding opportunities for exactly two built-in functions -- just to make the functions tempting enough to make people more liable to use them -- the fact that the proposal locks down some syntax forever to mean this one thing, is something I really would expect to see justified.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @sobolevn We may need to revise this section. Thank you @eli-schwartz There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. And for context, an example of something the PEP could argue against -- the PEP I'd probably write, if I had the time / sponsorship -- would be to solve the problem of "builtins cannot be unambiguously referenced after being overwritten / reused as a kwarg, and even The obvious thing that springs to mind for using f{}, aside for "f-sets", is (This is basically why I dislike the PEP itself -- it feels like "thinking small" / not very imaginative about pushing the boundaries of python.)
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @eli-schwartz this is not really a place for technical discusions, let's please move to https://discuss.python.org/t/pep-841-adding-frozen-syntax-to-make-immutable-types-optimizable/108219/32 for better visibility :) |
||||||||||
| * Bracket variants: ``{{'key': 1}}``, ``|{'key': 1}|``, ``{|'key': 1|}``. | ||||||||||
| This was rejected because adding | ||||||||||
| a single token ``f{`` is easier than adding two tokens. | ||||||||||
| Writing ``f{`` is also easier than writing two different brackets. | ||||||||||
| ``{{}}`` is rejected because it is a valid syntax right now. | ||||||||||
| * Word prefixes: ``frozen{'key': 1}``, ``fdict{'key': 1}``, | ||||||||||
| ``frozendict {'key': 1}``. | ||||||||||
| This was rejected because it is rather verbose. | ||||||||||
|
|
||||||||||
| We also rejected using ``F{`` as it is possible with fstrings. | ||||||||||
| This was rejected to keep the syntax as minimalistic as possible | ||||||||||
| and not to create extra ``F{`` token. | ||||||||||
|
|
||||||||||
| Freezing methods | ||||||||||
| ---------------- | ||||||||||
|
|
||||||||||
| Methods such as ``{'key': 1}.freeze()`` or | ||||||||||
| ``{'key': 1}.take_frozendict()`` are not real alternatives: they can be | ||||||||||
| added independently of this PEP. | ||||||||||
|
|
||||||||||
| While such methods can be a great feature on its own, | ||||||||||
| making this the only way to create immutable containers is not an option: | ||||||||||
|
|
||||||||||
| * Multiline expressions with such methods are really hard to read, | ||||||||||
| because you need to read the very last line to know the type of the object. | ||||||||||
| * It is quite verbose to write for common cases. | ||||||||||
| * It does not provide syntax guarantees for static analysis tools. | ||||||||||
| * It may have a different semantics when used | ||||||||||
| as ``a = {1: 2}; b(a.take_frozenset())``, | ||||||||||
| dependending on `how the internals would look like <https://github.com/capi-workgroup/decisions/issues/109>`_, | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
| it might mean that ``a`` would be cleared. | ||||||||||
|
|
||||||||||
|
|
||||||||||
| Copyright | ||||||||||
| ========= | ||||||||||
|
|
||||||||||
| This document is placed in the public domain or under the | ||||||||||
| CC0-1.0-Universal license, whichever is more permissive. | ||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps?
Or:
Sentence case versions:
Or: