Guard the no-telemetry promise in the code, not just in the prose - #175
Merged
Conversation
…in the prose
Convention 36 was stated in four places and checked in none. The two
tests that mention telemetry both check that we SAY it. For a tool that
intercepts the user's traffic, "it sends nothing anywhere" is the one
promise where being wrong is not a bug.
Two layers, because neither contains the other:
- static (AST) sees code that no test executes, and this package has
such code: host_identity's gethostbyname fallback never runs on an
ordinary path
- runtime (audit hook, PEP 578, in a subprocess) sees what is not
written down literally. Measured on 3.14.7: ctypes.dlopen carries the
library name even when computed, socket.connect carries the
destination, and the import event catches __import__("url" + "lib")
The allowlists are measured, not guessed. A CLI --simulate run with
impairment armed raises zero network events and zero forbidden imports;
building the whole GUI raises zero; host_identity raises exactly five,
all of them the documented route probe. So the guard asserts where a
connection goes, not merely that a socket appeared.
Eight mutations on the real tree, 8/8 caught, three of them added to the
mutation registry so CI repeats them. A permanent canary points the
scanner at seven shapes it must reject and one clean case it must not.
What it cannot prove is in the docstring rather than implied: a C
extension calling WinSock directly raises no socket event, the runtime
layer only sees what it runs, and a computed import inside a function
nothing calls is invisible to both layers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured in WSL: the socket events and their destinations fire the same on both runners, but dlopen comes back empty there, so the runtime library check gives nothing on the Ubuntu leg. The static check is the one that covers it everywhere, and a green run that measures nothing has to say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The guard shipped saying the overhead was not measured, which is how a sentence like that survives for years. Two full runs on a bench that refuses a verdict when the difference sits inside the noise. An installed hook charges ~235-310 ns per audit event. The ratio repeated (4.66x and 4.76x on a dispatch-only workload); the absolute difference did not (0.118 s and 0.154 s), so the range is the honest form and neither absolute should be quoted as the number. This test raises 2010 events, roughly half a millisecond, and the bench refused a conclusion on its workload in both runs: against a 1.2 s run whose own spread is 50 to 200 ms the hook cannot be seen. What the test costs is the second that --duration 1 waits out. The first version of the bench looped on open() and was refused on every column, correctly - opening a file 20000 times measures file work, not dispatch, and its own spread was ten times any hook cost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An outside review pointed at the shape and measurement confirmed it: the scanner matched socket.x(...) literally, so five ways of doing the same thing walked straight through. Each was fed to the scanner rather than reasoned about: - import socket as s / s.create_connection(...) - from socket import create_connection / create_connection(...) - import subprocess as sp / sp.run(...) - from subprocess import Popen / Popen(...) - os.popen(...), which was simply missing from the spawn list The review was wrong about socket.send/sendall/sendto: those need a socket, and a socket needs socket.socket or socket.create_connection, which were already gated. The gate sits at creation, which is one place instead of every method a socket has. That reasoning now lives next to the registry instead of only in my head. Measuring it turned up a defect the review did not: a module carrying the literal "://" CRASHED the guard on an IndexError. A scanner that dies is worse than one that misses. The fix has two halves and the second is load-bearing: local names are resolved against what the module imported, and the import itself is registered per file. A statement names its module whatever the local name becomes, so f = socket.create_connection; f(...) - which defeats any call-site matching - is caught at the import. Also: the from-ctypes-import loader form, and the canary grown from seven cases to seventeen, one per spelling. Verified: 9/9 mutations on the real tree including three new ones, all four telemetry entries caught through tools/mutate.py, ten tests green on Windows and in WSL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Convention 36 - the shipped program reaches no network - was stated in four places and
checked in none.
test_license_surface.pyandtest_windows.pyboth verify that we SAYit. Nothing had ever looked at the code. For a tool that intercepts the user's traffic,
"it sends nothing anywhere" is the one promise where being wrong is not a bug.
Two layers, because neither contains the other
The obvious answer is an AST scan. That was the first design, and it picked the wrong
primitive: the interpreter already answers this question a layer down, through PEP 578
audit hooks. Measured on CPython 3.14.7:
import urllib.request__import__("url" + "lib.request")ctypes.windll.wininetSo the static half answers "was it written" and the runtime half answers "what actually
happened". Dropping either leaves a whole shape of bypass open, which is why both ship.
The allowlists are measured, not guessed
--simulaterun with impairment armed: zero network events, zero forbiddenimports, four local libraries
host_identity(): exactly five events, all of them the documented route probeBecause the baseline is zero, the guard can assert the strongest shape there is: nothing,
except four entries that each carry a written reason. And because the audit event carries
the destination, it asserts
8.8.8.8:80and2001:4860:4860::8888:80rather than merelythat a socket appeared - "a socket happened" would pass for a socket that shipped the
user's captured traffic somewhere.
What is checked
Static: no network client imported;
socketandwebbrowserused only where a registrynames a reason;
ctypeslimited to the ten local Windows libraries; no process spawned(so no shelling out to curl or PowerShell); URL literals only in
appinfo.pyandlegal.py, with docstrings exempt because prose about a URL is not a URL.Runtime: no unexpected network event, no forbidden import, no unexpected library, and
every
connectnaming a documented probe address.The exception registry is exact in both directions. An entry naming code that has
moved fails like a call with no entry - permission nobody can point at any more is how an
allowlist becomes a blindfold.
Proof
Eight mutations against the real tree, 8/8 caught. Three are added to the mutation
registry so CI repeats them: a plain import, a network library through
ctypes, and aclient imported under a computed name at runtime.
A permanent canary points the scanner at seven shapes it must reject plus one clean case
it must not, modelled on
tests/test_public_text_guard.py- the file that earned thisproject the lesson that a guard nobody has watched fail is indistinguishable from a guard
that reads nothing.
What it cannot prove, in the docstring rather than implied
socketevent.psutilis such anextension: the hook covers third-party Python, not third-party C.
one was found by mutating rather than by thinking, and moving the same call onto a path
the CLI runs gets it caught at once.
but nothing loads a Windows library there, and the static half is what covers it.
This is a lock on the surface, not a proof of silence, and it says so.
Two findings the measurement produced
utils.host_identityhas asocket.gethostbynamefallback that convention 36 nevermentioned, and a name resolution can put a query on the wire - the convention's reasoning
("a UDP connect sends no packet") does not cover it. Measured: it never runs here, because
the route probe answers first. It stays as an exception, to be measured another time, and
it is named in the registry with that reason rather than passed over silently.
Separately, the convention says "one
socket.socketinbeantester/" - true of the calls,but
import socketis in two modules (settings.pyreads the machine's own services fileto label a port). A guard written from the convention alone would have had the wrong
allowlist on day one.
Checks
test_no_telemetry.py,test_mutation_registry.py,test_repo_conventions.pyandtest_code_hygiene.pypass, plus ruff and mypy on the new file. Run on Windows and, forthe platform question above, in WSL.
🤖 Generated with Claude Code