Skip to content

gh-118150: difflib: expose autojunk flag from SequenceMatcher to public methods and functions #153959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
faithlesstomas wants to merge 9 commits into
base: main
Choose a base branch
from
+41 −17
Open

gh-118150: difflib: expose autojunk flag from SequenceMatcher to public methods and functions #153959

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
367dc55
difflib: expose autojunk flag from SequenceMatcher to public methods/…
faithlesstomas Jul 18, 2026
9a822c4
difflib: update news for gh118150
faithlesstomas Jul 18, 2026
5e2f622
difflib: fix typyos in docstrings
faithlesstomas Jul 18, 2026
7d86473
gh-118150: difflib - fix NEWS issue with ref. not found
faithlesstomas Jul 18, 2026
b5213c9
gh-118150: difflib - fix NEWS doc for ReST
faithlesstomas Jul 19, 2026
7d7a35b
gh-118150: difflib - remove template notes from NEWS
faithlesstomas Jul 20, 2026
bd98298
gh-118150: CI fix for check-docs
faithlesstomas Jul 22, 2026
af14a4f
gh-118150: how to reference a class in NEWS file?
faithlesstomas Jul 22, 2026
e3d69d8
Merge branch 'main' into gh-118150-expose-autojunk
faithlesstomas Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -656,7 +656,7 @@ jobs:
build-emscripten,
build-ubuntu-ssltests,
test-hypothesis,
cifuzz,
cifuzz

@hugovk hugovk Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this change for?

Suggested change
cifuzz
cifuzz,

allowed-skips: >-
${{ !fromJSON(needs.build-context.outputs.run-docs) && 'check-docs,' || '' }}
${{
Expand Down
46 changes: 30 additions & 16 deletions Lib/difflib.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -664,7 +664,7 @@ def real_quick_ratio(self):
__class_getitem__ = classmethod(GenericAlias)


def get_close_matches(word, possibilities, n=3, cutoff=0.6):
def get_close_matches(word, possibilities, n=3, cutoff=0.6, autojunk=True):

@hugovk hugovk Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we make this keyword only? It can be useful as the list gets longer, and especially for Booleans.

Suggested change
def get_close_matches(word, possibilities, n=3, cutoff=0.6, autojunk=True):
def get_close_matches(word, possibilities, n=3, cutoff=0.6, *, autojunk=True):

Also update the docstring.

"""Use SequenceMatcher to return list of the best "good enough" matches.

word is a sequence for which close matches are desired (typically a
Expand Down Expand Up @@ -698,7 +698,7 @@ def get_close_matches(word, possibilities, n=3, cutoff=0.6):
if not 0.0 <= cutoff <= 1.0:
raise ValueError("cutoff must be in [0.0, 1.0]: %r" % (cutoff,))
result = []
s = SequenceMatcher()
s = SequenceMatcher(autojunk=autojunk)
s.set_seq2(word)
for x in possibilities:
s.set_seq1(x)
Expand Down Expand Up @@ -810,7 +810,7 @@ class Differ:
+ 5. Flat is better than nested.
"""

def __init__(self, linejunk=None, charjunk=None):
def __init__(self, linejunk=None, charjunk=None, autojunk=True):

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here?

Suggested change
def __init__(self, linejunk=None, charjunk=None, autojunk=True):
def __init__(self, linejunk=None, charjunk=None, *, autojunk=True):

"""
Construct a text differencer, with optional filters.

Expand All @@ -828,10 +828,13 @@ def __init__(self, linejunk=None, charjunk=None):
module-level function `IS_CHARACTER_JUNK` may be used to filter out
whitespace characters (a blank or tab; **note**: bad idea to include
newline in this!). Use of IS_CHARACTER_JUNK is recommended.
- `autojunk`: automatic junk diff heuristic
(refer to :class:`SequenceMatcher` for specifics).
"""

self.linejunk = linejunk
self.charjunk = charjunk
self.autojunk = autojunk

def compare(self, a, b):
r"""
Expand Down Expand Up @@ -859,7 +862,7 @@ def compare(self, a, b):
+ emu
"""

cruncher = SequenceMatcher(self.linejunk, a, b)
cruncher = SequenceMatcher(self.linejunk, a, b, autojunk=self.autojunk)
for tag, alo, ahi, blo, bhi in cruncher.get_opcodes():
if tag == 'replace':
g = self._fancy_replace(a, alo, ahi, b, blo, bhi)
Expand Down Expand Up @@ -920,7 +923,7 @@ def _fancy_replace(self, a, alo, ahi, b, blo, bhi):
# Later, more pathological cases prompted removing recursion
# entirely.
cutoff = 0.74999
cruncher = SequenceMatcher(self.charjunk)
cruncher = SequenceMatcher(self.charjunk, autojunk=self.autojunk)
crqr = cruncher.real_quick_ratio
cqr = cruncher.quick_ratio
cr = cruncher.ratio
Expand Down Expand Up @@ -1099,7 +1102,7 @@ def _format_range_unified(start, stop):
return '{},{}'.format(beginning, length)

def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
tofiledate='', n=3, lineterm='\n', *, color=False):
tofiledate='', n=3, lineterm='\n', *, color=False, autojunk=True):

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are already keyword-only, and don't have any special order, so could alphabetise? If so, update docstring to match.

Suggested change
tofiledate='', n=3, lineterm='\n', *, color=False, autojunk=True):
tofiledate='', n=3, lineterm='\n', *, autojunk=True, color=False):

r"""
Compare two sequences of lines; generate the delta as a unified diff.

Expand All @@ -1120,6 +1123,9 @@ def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
'git diff --color'. Even if enabled, it can be
controlled using environment variables such as 'NO_COLOR'.

Set `autojunk` to False if you don't want automated junk heuristic.
See details in :class:`SequenceMatcher.

The unidiff format normally has a header for filenames and modification
times. Any or all of these may be specified using strings for
'fromfile', 'tofile', 'fromfiledate', and 'tofiledate'.
Expand Down Expand Up @@ -1150,7 +1156,7 @@ def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',

_check_types(a, b, fromfile, tofile, fromfiledate, tofiledate, lineterm)
started = False
for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
for group in SequenceMatcher(None,a,b,autojunk=autojunk).get_grouped_opcodes(n):

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for group in SequenceMatcher(None,a,b,autojunk=autojunk).get_grouped_opcodes(n):
for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

if not started:
started = True
fromdate = '\t{}'.format(fromfiledate) if fromfiledate else ''
Expand Down Expand Up @@ -1193,7 +1199,7 @@ def _format_range_context(start, stop):

# See http://www.unix.org/single_unix_specification/
def context_diff(a, b, fromfile='', tofile='',
fromfiledate='', tofiledate='', n=3, lineterm='\n'):
fromfiledate='', tofiledate='', n=3, lineterm='\n', autojunk=True):

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here?

Suggested change
fromfiledate='', tofiledate='', n=3, lineterm='\n', autojunk=True):
fromfiledate='', tofiledate='', n=3, lineterm='\n', *, autojunk=True):

r"""
Compare two sequences of lines; generate the delta as a context diff.

Expand All @@ -1216,6 +1222,10 @@ def context_diff(a, b, fromfile='', tofile='',
The modification times are normally expressed in the ISO 8601 format.
If not specified, the strings default to blanks.

The kwarg `autojunk` sets up automated junk heuristic with
:class:`SequenceMatcher`, which is used under the hood in this function.
See documentation of :class:`SequenceMatcher` for details.

Example:

>>> print(''.join(context_diff('one\ntwo\nthree\nfour\n'.splitlines(True),
Expand All @@ -1239,7 +1249,7 @@ def context_diff(a, b, fromfile='', tofile='',
_check_types(a, b, fromfile, tofile, fromfiledate, tofiledate, lineterm)
prefix = dict(insert='+ ', delete='- ', replace='! ', equal=' ')
started = False
for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
for group in SequenceMatcher(None,a,b, autojunk=autojunk).get_grouped_opcodes(n):

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for group in SequenceMatcher(None,a,b, autojunk=autojunk).get_grouped_opcodes(n):
for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

if not started:
started = True
fromdate = '\t{}'.format(fromfiledate) if fromfiledate else ''
Expand Down Expand Up @@ -1321,7 +1331,7 @@ def decode(s):
for line in lines:
yield line.encode('ascii', 'surrogateescape')

def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK):
def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, autojunk=True):

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, autojunk=True):
def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, *, autojunk=True):

r"""
Compare `a` and `b` (lists of strings); return a `Differ`-style delta.

Expand All @@ -1339,6 +1349,8 @@ def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK):
whitespace characters (a blank or tab; note: it's a bad idea to
include newline in this!).

- autojunk: automatic junk heuristic - refer to :class:`SequenceMatcher` for details

Tools/scripts/ndiff.py is a command-line front-end to this function.

Example:
Expand All @@ -1356,10 +1368,10 @@ def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK):
+ tree
+ emu
"""
return Differ(linejunk, charjunk).compare(a, b)
return Differ(linejunk, charjunk, autojunk).compare(a, b)

def _mdiff(fromlines, tolines, context=None, linejunk=None,
charjunk=IS_CHARACTER_JUNK):
charjunk=IS_CHARACTER_JUNK, autojunk=True):
r"""Returns generator yielding marked up from/to side by side differences.

Arguments:
Expand All @@ -1369,6 +1381,7 @@ def _mdiff(fromlines, tolines, context=None, linejunk=None,
if None, all from/to text lines will be generated.
linejunk -- passed on to ndiff (see ndiff documentation)
charjunk -- passed on to ndiff (see ndiff documentation)
autojunk -- passed on to ndiff (see ndiff documentation)

This function returns an iterator which returns a tuple:
(from line tuple, to line tuple, boolean flag)
Expand Down Expand Up @@ -1398,7 +1411,7 @@ def _mdiff(fromlines, tolines, context=None, linejunk=None,
change_re = re.compile(r'(\++|\-+|\^+)')

# create the difference iterator to generate the differences
diff_lines_iterator = ndiff(fromlines,tolines,linejunk,charjunk)
diff_lines_iterator = ndiff(fromlines, tolines, linejunk, charjunk, autojunk)

def _make_line(lines, format_key, side, num_lines=[0,0]):
"""Returns line of text with user's change markup and line formatting.
Expand Down Expand Up @@ -1738,21 +1751,22 @@ class HtmlDiff(object):
_default_prefix = 0

def __init__(self,tabsize=8,wrapcolumn=None,linejunk=None,
charjunk=IS_CHARACTER_JUNK):
charjunk=IS_CHARACTER_JUNK, autojunk=True):
"""HtmlDiff instance initializer

Arguments:
tabsize -- tab stop spacing, defaults to 8.
wrapcolumn -- column number where lines are broken and wrapped,
defaults to None where lines are not wrapped.
linejunk,charjunk -- keyword arguments passed into ndiff() (used by
linejunk, charjunk, autojunk -- keyword arguments passed into ndiff() (used by
HtmlDiff() to generate the side by side HTML differences). See
ndiff() documentation for argument default values and descriptions.
"""
self._tabsize = tabsize
self._wrapcolumn = wrapcolumn
self._linejunk = linejunk
self._charjunk = charjunk
self._autojunk = autojunk

def make_file(self, fromlines, tolines, fromdesc='', todesc='',
context=False, numlines=5, *, charset='utf-8'):
Expand Down Expand Up @@ -2026,7 +2040,7 @@ def make_table(self,fromlines,tolines,fromdesc='',todesc='',context=False,
else:
context_lines = None
diffs = _mdiff(fromlines,tolines,context_lines,linejunk=self._linejunk,
charjunk=self._charjunk)
charjunk=self._charjunk, autojunk=self._autojunk)

# set up iterator to wrap lines that exceed desired width
if self._wrapcolumn:
Expand Down
10 changes: 10 additions & 0 deletions Misc/NEWS.d/next/Library/2026-07-18-16-43-02.gh-issue-118150.x0mV8P.rst
View file
Open in desktop

@Lenormju Lenormju Jul 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@faithlesstomas you should review this file, it seems to have been very badly AI-generated.

@faithlesstomas faithlesstomas Jul 19, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lenormju I wouldn't have thought that my bad English could be considered to be AI generated :) Do you have any specific suggestion what should be improved in this description?

BTW, nothing in this PR was AI generated.

@Lenormju Lenormju Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@faithlesstomas The commit was titled difflib - fix NEWS issue with ref. not found which I found strange that a human would commit "ref. not found". Also the NEWS file contain these comments :

# # Uncomment one of these "section:" lines to specify which section # this
entry should go in in Misc/NEWS.d. # #.. section: Security #.. section: Core
and Builtins #.. section: Library #.. section: Documentation #.. section:
Tests #.. section: Build #.. section: Windows #.. section: macOS #..
section: IDLE #.. section: Tools/Demos #.. section: C API

# Write your Misc/NEWS.d entry below.  It should be a simple ReST paragraph.
# Don't start with "- Issue #<n>: " or "- gh-issue-<n>: " or that sort of
stuff.
###########################################################################

Which look like a template you did not erase.

I was not basing myself on the English, instead on all the rest. But if it wasn't, sorry.

@faithlesstomas faithlesstomas Jul 22, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lenormju Sorry, that was my mistake - I thought this comment is to be left there. Also I didn't know how properly reference a class in ReST... But besides that, does this PR look good now?

@hugovk hugovk Jul 24, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blurb CLI uses .. section: Library to decide which directory to put the news file, in this case it's already in Misc/NEWS.d/next/Library/ as expected.

Please remove .. section: Library and dedent the next paragraph. News files are usually fairly short, often just a line or two, so we could do the same here.

See some other examples in https://github.com/python/cpython/tree/main/Misc/NEWS.d/next/Library

Please also add a What's New entry to https://github.com/python/cpython/blob/main/Doc/whatsnew/3.16.rst, this can be a bit longer if needed.

Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Expose automated junk heruistic flag to the public in difflib.

.. section: Library
This change exposes the *autojunk* kwarg — used for turning on/off
the automated junk heuristic in the :class:`difflib.SequenceMatcher`
(which defaults to ``True``, meaning this heuristic was always performed
in :mod:`difflib`) — to the public methods and functions of
the :mod:`difflib`, so that users can change the behavior of this option.
See the :class:`difflib.SequenceMatcher` documentation for details.
Loading