gh-118150: difflib: expose autojunk flag from SequenceMatcher to public methods and functions by faithlesstomas · Pull Request #153959 · python/cpython

faithlesstomas · 2026-07-18T13:27:56Z

Until now difflib methods and functions took SequenceMatcher class kwarg autojunk by default which is set up to be True in this class.. This PR aims to expose this flag to the public methods and functions of this module, in order that user can choose behavior of this option in SequenceMatcher.

Issue: #118150
Related PR: #153892

Issue: Difflib creates unreasonably large diffs #118150

…functions

python-cla-bot · 2026-07-18T13:28:00Z

All commit authors signed the Contributor License Agreement.

bedevere-app · 2026-07-18T13:28:00Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

faithlesstomas · 2026-07-18T13:48:33Z

Quick check if this works for the original issue test:

 #!/usr/bin/python3                                                                                                                                                          
                                                                                                                                                                              
  import difflib                                                                                                                                                              
                                                                                                                                                                              
  def get_lines(filename):                                                                                                                                                    
      with open(filename, 'r', encoding='utf8') as fd:                                                                                                                        
          return fd.readlines()                                                                                                                                               
                                                                                                                                                                              
  for autojunk in (True, False):                                                                                                                                              
                                                                                                                                                                              
      old_new = list(difflib.unified_diff(                                                                                                                                    
          get_lines('small.external.old.json'),                                                                                                                               
          get_lines('small.external.new.json'),                                                                                                                               
          autojunk=autojunk                                                                                                                                                   
      ))                                                                                                                                                                      
      new_old = list(difflib.unified_diff(                                                                                                                                    
          get_lines('small.external.new.json'),                                                                                                                               
          get_lines('small.external.old.json'),                                                                                                                               
          autojunk=autojunk                                                                                                                                                   
      ))                                                                                                                                                                      
                                                                                                                                                                              
      print('Autojunk flag:', autojunk)                                                                                                                                       
      print('diff external.old external.new.json | wc -l')                                                                                                                    
      print(len(old_new))                                                                                                                                                     
      print('diff external.new external.old.json | wc -l')                                                                                                                    
      print(len(new_old))

Autojunk flag: True
diff external.old external.new.json | wc -l
90
diff -u external.new external.old.json | wc -l
25
Autojunk flag: False
diff external.old external.new.json | wc -l
25
diff -u external.new external.old.json | wc -l
25

#118150 (comment)

Lenormju · 2026-07-18T15:27:36Z

I think the Documentation (Doc/library/difflib.rst) should get updated too to include this new parameter.

Lenormju · 2026-07-18T15:43:22Z

@faithlesstomas you should review this file, it seems to have been very badly AI-generated.

@Lenormju I wouldn't have thought that my bad English could be considered to be AI generated :) Do you have any specific suggestion what should be improved in this description?

BTW, nothing in this PR was AI generated.

@faithlesstomas The commit was titled difflib - fix NEWS issue with ref. not found which I found strange that a human would commit "ref. not found". Also the NEWS file contain these comments :

# # Uncomment one of these "section:" lines to specify which section # this entry should go in in Misc/NEWS.d. # #.. section: Security #.. section: Core and Builtins #.. section: Library #.. section: Documentation #.. section: Tests #.. section: Build #.. section: Windows #.. section: macOS #.. section: IDLE #.. section: Tools/Demos #.. section: C API # Write your Misc/NEWS.d entry below. It should be a simple ReST paragraph. # Don't start with "- Issue #<n>: " or "- gh-issue-<n>: " or that sort of stuff. ###########################################################################

Which look like a template you did not erase.

I was not basing myself on the English, instead on all the rest. But if it wasn't, sorry.

@Lenormju Sorry, that was my mistake - I thought this comment is to be left there. Also I didn't know how properly reference a class in ReST... But besides that, does this PR look good now?

The blurb CLI uses .. section: Library to decide which directory to put the news file, in this case it's already in Misc/NEWS.d/next/Library/ as expected.

Please remove .. section: Library and dedent the next paragraph. News files are usually fairly short, often just a line or two, so we could do the same here.

See some other examples in https://github.com/python/cpython/tree/main/Misc/NEWS.d/next/Library

Please also add a What's New entry to https://github.com/python/cpython/blob/main/Doc/whatsnew/3.16.rst, this can be a bit longer if needed.

hugovk

Please could you add tests to Lib/test/test_difflib.py?

hugovk · 2026-07-24T09:56:05Z

          build-ubuntu-ssltests,
          test-hypothesis,
-          cifuzz,
+          cifuzz


What's this change for?

Suggested change

cifuzz

cifuzz,

hugovk · 2026-07-24T09:57:25Z



-def get_close_matches(word, possibilities, n=3, cutoff=0.6):
+def get_close_matches(word, possibilities, n=3, cutoff=0.6, autojunk=True):


Shall we make this keyword only? It can be useful as the list gets longer, and especially for Booleans.

Suggested change

def get_close_matches(word, possibilities, n=3, cutoff=0.6, autojunk=True):

def get_close_matches(word, possibilities, n=3, cutoff=0.6, *, autojunk=True):

Also update the docstring.

hugovk · 2026-07-24T09:58:54Z

    """

-    def __init__(self, linejunk=None, charjunk=None):
+    def __init__(self, linejunk=None, charjunk=None, autojunk=True):


And here?

Suggested change

def __init__(self, linejunk=None, charjunk=None, autojunk=True):

def __init__(self, linejunk=None, charjunk=None, *, autojunk=True):

hugovk · 2026-07-24T10:00:05Z


 def unified_diff(a, b, fromfile='', tofile='', fromfiledate='',
-                 tofiledate='', n=3, lineterm='\n', *, color=False):
+                 tofiledate='', n=3, lineterm='\n', *, color=False, autojunk=True):


These are already keyword-only, and don't have any special order, so could alphabetise? If so, update docstring to match.

Suggested change

tofiledate='', n=3, lineterm='\n', *, color=False, autojunk=True):

tofiledate='', n=3, lineterm='\n', *, autojunk=True, color=False):

hugovk · 2026-07-24T10:01:06Z

    _check_types(a, b, fromfile, tofile, fromfiledate, tofiledate, lineterm)
    started = False
-    for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
+    for group in SequenceMatcher(None,a,b,autojunk=autojunk).get_grouped_opcodes(n):


Suggested change

for group in SequenceMatcher(None,a,b,autojunk=autojunk).get_grouped_opcodes(n):

for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

hugovk · 2026-07-24T10:01:22Z

 # See http://www.unix.org/single_unix_specification/
 def context_diff(a, b, fromfile='', tofile='',
-                 fromfiledate='', tofiledate='', n=3, lineterm='\n'):
+                 fromfiledate='', tofiledate='', n=3, lineterm='\n', autojunk=True):


And here?

Suggested change

fromfiledate='', tofiledate='', n=3, lineterm='\n', autojunk=True):

fromfiledate='', tofiledate='', n=3, lineterm='\n', *, autojunk=True):

hugovk · 2026-07-24T10:01:32Z

        yield line.encode('ascii', 'surrogateescape')

-def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK):
+def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, autojunk=True):


Suggested change

def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, autojunk=True):

def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, *, autojunk=True):

hugovk · 2026-07-24T10:06:23Z

The blurb CLI uses .. section: Library to decide which directory to put the news file, in this case it's already in Misc/NEWS.d/next/Library/ as expected.

Please remove .. section: Library and dedent the next paragraph. News files are usually fairly short, often just a line or two, so we could do the same here.

See some other examples in https://github.com/python/cpython/tree/main/Misc/NEWS.d/next/Library

Please also add a What's New entry to https://github.com/python/cpython/blob/main/Doc/whatsnew/3.16.rst, this can be a bit longer if needed.

hugovk · 2026-07-24T10:15:37Z

    prefix = dict(insert='+ ', delete='- ', replace='! ', equal='  ')
    started = False
-    for group in SequenceMatcher(None,a,b).get_grouped_opcodes(n):
+    for group in SequenceMatcher(None,a,b, autojunk=autojunk).get_grouped_opcodes(n):


Suggested change

for group in SequenceMatcher(None,a,b, autojunk=autojunk).get_grouped_opcodes(n):

for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

difflib: expose autojunk flag from SequenceMatcher to public methods/…

367dc55

…functions

bedevere-app Bot added the awaiting review label Jul 18, 2026

tomasr8 added the sprint label Jul 18, 2026

github-project-automation Bot moved this to Todo in Sprint Jul 18, 2026

github-project-automation Bot added this to Sprint Jul 18, 2026

faithlesstomas changed the title ~~difflib: expose autojunk flag from SequenceMatcher to public methods and functions~~ gh-118150: difflib: expose autojunk flag from SequenceMatcher to public methods and functions Jul 18, 2026

bedevere-app Bot mentioned this pull request Jul 18, 2026

Difflib creates unreasonably large diffs #118150

Open

This was referenced Jul 18, 2026

SequenceMatcher & autojunk - false negative #90825

Open

gh-118150: warn in doc about results asymmetry for difflib junk #153892

Open

faithlesstomas added 3 commits July 18, 2026 16:44

difflib: update news for gh118150

9a822c4

difflib: fix typyos in docstrings

5e2f622

pythongh-118150: difflib - fix NEWS issue with ref. not found

7d86473

Lenormju reviewed Jul 18, 2026

View reviewed changes

faithlesstomas added 3 commits July 19, 2026 11:31

pythongh-118150: difflib - fix NEWS doc for ReST

b5213c9

pythongh-118150: difflib - remove template notes from NEWS

7d7a35b

pythongh-118150: CI fix for check-docs

bd98298

faithlesstomas requested review from AA-Turner, JacobCoffee, ezio-melotti, hugovk, itamaro and webknjaz as code owners July 22, 2026 11:44

faithlesstomas and others added 2 commits July 22, 2026 13:56

pythongh-118150: how to reference a class in NEWS file?

af14a4f

Merge branch 'main' into pythongh-118150-expose-autojunk

e3d69d8

hugovk reviewed Jul 24, 2026

View reviewed changes



		def get_close_matches(word, possibilities, n=3, cutoff=0.6):
		def get_close_matches(word, possibilities, n=3, cutoff=0.6, autojunk=True):

	def __init__(self, linejunk=None, charjunk=None, autojunk=True):
	def __init__(self, linejunk=None, charjunk=None, *, autojunk=True):

	tofiledate='', n=3, lineterm='\n', *, color=False, autojunk=True):
	tofiledate='', n=3, lineterm='\n', *, autojunk=True, color=False):

	for group in SequenceMatcher(None,a,b,autojunk=autojunk).get_grouped_opcodes(n):
	for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

	fromfiledate='', tofiledate='', n=3, lineterm='\n', autojunk=True):
	fromfiledate='', tofiledate='', n=3, lineterm='\n', *, autojunk=True):

	def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, autojunk=True):
	def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK, *, autojunk=True):

	for group in SequenceMatcher(None,a,b, autojunk=autojunk).get_grouped_opcodes(n):
	for group in SequenceMatcher(None, a, b, autojunk=autojunk).get_grouped_opcodes(n):

Uh oh!

Uh oh!

Conversation

faithlesstomas commented Jul 18, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot Bot commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app Bot commented Jul 18, 2026

Uh oh!

faithlesstomas commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lenormju commented Jul 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

hugovk Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hugovk Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

faithlesstomas commented Jul 18, 2026 •

edited by bedevere-app Bot

Loading

python-cla-bot Bot commented Jul 18, 2026 •

edited

Loading

faithlesstomas commented Jul 18, 2026 •

edited

Loading

hugovk Jul 24, 2026 •

edited

Loading

hugovk Jul 24, 2026 •

edited

Loading