Enforce mutation conflicts in both directions when building chain lists - #3
Open
VOE9 wants to merge 1 commit into
Open
Enforce mutation conflicts in both directions when building chain lists#3VOE9 wants to merge 1 commit into
VOE9 wants to merge 1 commit into
Conversation
combinatedOpListFactory() decides whether a candidate operator may join an
existing chain by consulting only the candidate's own __exclusion_op__
list: it asks "does the new operator exclude anything already in the
chain?" but never "does anything already in the chain exclude the new
operator?".
The exclusion tables are hand-written and are not symmetric -- 910 pairs
(A, B) exist where A's list names B but B's list does not name A. For
those pairs the one-directional check admits the combination whenever the
operator whose list is incomplete happens to be added last, so chains that
the conflict tables are meant to forbid are still emitted.
Concrete example: M04_PDF ("Change Extensions to Mutation Resource File")
lists M07 ("Remove extension"), but M07 does not list M04_PDF. Both revise
the same portion of the request -- the file extension -- so a chain
containing both sets the extension to .pdf and removes it at the same
time. On the js seed, 336 generated chains contain both operators.
This matches the algorithm the NDSS'20 paper describes in Section V-B and
Section VII: a conflicting mutation is defined as one where "both M1 and
M2 revise the same portion of a mutation vector", and "we discarded a
combination in which one of its mutation operations conflicted with other
mutation operations". Overlap is symmetric by construction, and the
chains are sorted (map(list.sort, ...)) so they carry no ordering that an
order-dependent check could rely on. The sibling routine mutation_chain()
happens to check the opposite direction, so the two paths currently
disagree with each other.
Fix: after the existing check, also verify that no operator already in the
chain excludes the candidate.
Measured with the project's own code (python2), counting chains that
contain a pair (a, b) where a's __exclusion_op__[seedtype] names b:
seed chains before violating before chains after violating after
js 21,743 12,144 (55.9%) 9,599 0
php 174,655 83,968 (48.1%) 90,687 0
html 27,239 13,984 (51.3%) 13,255 0
xhtml 4,975 1,824 (36.7%) 3,151 0
Note this removes about half of the generated chains. Every removed chain
is one that the conflict tables already marked as invalid, but it is a
visible change to the size of the search space, so please weigh it against
what you expect for the published evaluation.
I used AI assistance for this, but not entirely.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
combinatedOpListFactory()enforces the mutation-conflict tables in only one direction, so chains that those tables are meant to forbid are still emitted. On the four bundled seed types this affects 111,920 of 228,612 generated chains (49%).The bug
When deciding whether a candidate operator may join an existing chain, the function consults only the candidate's own
__exclusion_op__list:It never asks the reverse: does an operator already in the chain exclude the candidate?
That would be harmless if the tables were symmetric, but they are hand-written and are not. I extracted
__exclusion_op__from all 57 operator modules withast.literal_evaland found 910 pairs (A, B) where A's list names B but B's list does not name A. For each such pair the combination is correctly rejected when A is added last, and incorrectly accepted when B is added last.Worked example
__comment__M04_PDFM07M07M04_PDFBoth revise the same portion of the request — the file extension — so a chain containing both sets the extension to
.pdfand removes it at the same time. On thejsseed, 336 generated chains contain both operators; one of them is['M03_JPG', 'M04_PDF', 'M07'].Why I read this as a bug rather than intended asymmetry
I checked this against the paper before proposing anything, because halving the search space is not a change to make on a guess.
map(list.sort, round_templist)sorts each chain, so no ordering survives that an order-dependent (one-way) rule could be keyed on.mutation_chain()checksi in base_chain_import[ele].mOP.__exclusion_op__[seed_type]— the opposite direction fromcombinatedOpListFactory(). Neither checks both.Fix
Eleven lines: keep the existing check, and add the reverse one.
Measurements
Run with the project's own code under
python2. A chain counts as violating if it contains a pair(a, b)wherea.__exclusion_op__[seedtype]namesb.To reproduce, from the repository root:
Impact, stated plainly
This removes roughly half of the generated chains. Every removed chain is one the conflict tables already marked invalid, and the paper's stated algorithm would not have produced them — but it is a visible change to the size of the search space, so it is worth weighing against what you expect for the published evaluation. If you would rather keep the current behaviour, or fix the tables instead of the check, I am happy to close this or rework it.
mutation_chain()is left untouched. It checks the opposite direction, and on single-operator base chains for thejsseed I measured 0 violations in its unchecked direction, so I did not want to change behaviour there without evidence.I used AI assistance for this, but not entirely.