Skip to content

Enforce mutation conflicts in both directions when building chain lists - #3

Open
VOE9 wants to merge 1 commit into
WSP-LAB:masterfrom
VOE9:fix/symmetric-exclusion-check
Open

Enforce mutation conflicts in both directions when building chain lists#3
VOE9 wants to merge 1 commit into
WSP-LAB:masterfrom
VOE9:fix/symmetric-exclusion-check

Conversation

@VOE9

@VOE9 VOE9 commented Aug 27, 2026

Copy link
Copy Markdown

Summary

combinatedOpListFactory() enforces the mutation-conflict tables in only one direction, so chains that those tables are meant to forbid are still emitted. On the four bundled seed types this affects 111,920 of 228,612 generated chains (49%).

The bug

When deciding whether a candidate operator may join an existing chain, the function consults only the candidate's own __exclusion_op__ list:

for banop in self.op_dict[aop[0]].mOP.__exclusion_op__[seedtype]:
    if banop in opl:          # does the NEW operator exclude something in the chain?
        banflag = True
        break

It never asks the reverse: does an operator already in the chain exclude the candidate?

That would be harmless if the tables were symmetric, but they are hand-written and are not. I extracted __exclusion_op__ from all 57 operator modules with ast.literal_eval and found 910 pairs (A, B) where A's list names B but B's list does not name A. For each such pair the combination is correctly rejected when A is added last, and incorrectly accepted when B is added last.

Worked example

operator __comment__ lists the other?
M04_PDF Change Extensions to Mutation Resource File yes — names M07
M07 Remove extension no — omits M04_PDF

Both revise the same portion of the request — the file extension — so a chain containing both sets the extension to .pdf and removes it at the same time. On the js seed, 336 generated chains contain both operators; one of them is ['M03_JPG', 'M04_PDF', 'M07'].

Why I read this as a bug rather than intended asymmetry

I checked this against the paper before proposing anything, because halving the search space is not a change to make on a guess.

  1. §V-B describes the rule as mutual: "two different mutation operations to conflict with each other in the case that they revise the overlapping portions of a seed request". Overlap is symmetric by construction — if A's edits overlap B's, B's overlap A's.
  2. §VII (Mutation conflicts) states the discard rule as an any-pair test: "we defined a conflicting mutation (M2) as when (1) both M1 and M2 revise the same portion of a mutation vector""we discarded a combination in which one of its mutation operations conflicted with other mutation operations". That is exactly "reject the chain if any member conflicts with any other", which the one-directional check implements only half of.
  3. Chains are order-insensitive. map(list.sort, round_templist) sorts each chain, so no ordering survives that an order-dependent (one-way) rule could be keyed on.
  4. The two code paths disagree with each other. mutation_chain() checks i in base_chain_import[ele].mOP.__exclusion_op__[seed_type] — the opposite direction from combinatedOpListFactory(). Neither checks both.

Fix

Eleven lines: keep the existing check, and add the reverse one.

Measurements

Run with the project's own code under python2. A chain counts as violating if it contains a pair (a, b) where a.__exclusion_op__[seedtype] names b.

seed chains before violating before chains after violating after
js 21,743 12,144 (55.9%) 9,599 0
php 174,655 83,968 (48.1%) 90,687 0
html 27,239 13,984 (51.3%) 13,255 0
xhtml 4,975 1,824 (36.7%) 3,151 0
total 228,612 111,920 (49.0%) 116,692 0

To reproduce, from the repository root:

# check.py  --  run with: python2 check.py
import filemutator
m = filemutator.mutate_manager()
for st in ['js', 'php', 'html', 'xhtml']:
    chains = m.combinatedOpListFactory(st)
    bad = 0
    for chain in chains:
        for a in chain:
            excl = m.op_dict[a].mOP.__exclusion_op__.get(st, [])
            if any(b != a and b in excl for b in chain):
                bad += 1
                break
    print "%-6s chains=%-7d violating=%-7d (%.1f%%)" % (
        st, len(chains), bad, 100.0 * bad / len(chains) if chains else 0)

Impact, stated plainly

This removes roughly half of the generated chains. Every removed chain is one the conflict tables already marked invalid, and the paper's stated algorithm would not have produced them — but it is a visible change to the size of the search space, so it is worth weighing against what you expect for the published evaluation. If you would rather keep the current behaviour, or fix the tables instead of the check, I am happy to close this or rework it.

mutation_chain() is left untouched. It checks the opposite direction, and on single-operator base chains for the js seed I measured 0 violations in its unchecked direction, so I did not want to change behaviour there without evidence.

I used AI assistance for this, but not entirely.

combinatedOpListFactory() decides whether a candidate operator may join an
existing chain by consulting only the candidate's own __exclusion_op__
list: it asks "does the new operator exclude anything already in the
chain?" but never "does anything already in the chain exclude the new
operator?".

The exclusion tables are hand-written and are not symmetric -- 910 pairs
(A, B) exist where A's list names B but B's list does not name A. For
those pairs the one-directional check admits the combination whenever the
operator whose list is incomplete happens to be added last, so chains that
the conflict tables are meant to forbid are still emitted.

Concrete example: M04_PDF ("Change Extensions to Mutation Resource File")
lists M07 ("Remove extension"), but M07 does not list M04_PDF. Both revise
the same portion of the request -- the file extension -- so a chain
containing both sets the extension to .pdf and removes it at the same
time. On the js seed, 336 generated chains contain both operators.

This matches the algorithm the NDSS'20 paper describes in Section V-B and
Section VII: a conflicting mutation is defined as one where "both M1 and
M2 revise the same portion of a mutation vector", and "we discarded a
combination in which one of its mutation operations conflicted with other
mutation operations". Overlap is symmetric by construction, and the
chains are sorted (map(list.sort, ...)) so they carry no ordering that an
order-dependent check could rely on. The sibling routine mutation_chain()
happens to check the opposite direction, so the two paths currently
disagree with each other.

Fix: after the existing check, also verify that no operator already in the
chain excludes the candidate.

Measured with the project's own code (python2), counting chains that
contain a pair (a, b) where a's __exclusion_op__[seedtype] names b:

  seed    chains before   violating before   chains after   violating after
  js             21,743     12,144 (55.9%)          9,599                 0
  php           174,655     83,968 (48.1%)         90,687                 0
  html           27,239     13,984 (51.3%)         13,255                 0
  xhtml           4,975      1,824 (36.7%)          3,151                 0

Note this removes about half of the generated chains. Every removed chain
is one that the conflict tables already marked as invalid, but it is a
visible change to the size of the search space, so please weigh it against
what you expect for the published evaluation.

I used AI assistance for this, but not entirely.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants