Self-supervised link prediction and definition-corruption detection on a closed, self-referential lexicon (~2707 words, each defined only in terms of other words in the same list) — with no pretrained language models, no external embeddings, and no externally-assigned semantic labels anywhere in the primary pipeline.
Research release status: experiments 1–20, seeded result summaries, figures, tests, and a 500k-entry data-pipeline stress test are included. Before a public upload, document the provenance and redistribution terms of the bundled raw lexicon; see Data and artifact terms.
- Research paper — compact methods, results, claims, and limitations
- Reproducibility guide — environment and exact commands
- Results index — experiment-to-artifact map
- Data card and model card
- Contributing, citation, and license
- Publication checklist — three owner decisions before upload
The main finding is not that a small model has acquired general language understanding. It is narrower and experimentally useful: informed word rows enable closed-lexicon compatibility learning; contrastive retrieval, ordered character encoding, and definition content each contribute measurable signal; and controlled three-step knowledge is present even when a decoder trained only on shorter chains cannot freely express it. Length-matched training repairs that last bottleneck.
Can a neural network learn local/structural coherence within this closed lexicon — without externally-assigned semantic labels — in a way that demonstrably generalizes to hidden relations, rather than only reproducing density/degree?
This is explicitly not a claim that the model has general language understanding. See Scientific integrity / what this does and doesn't show below — none of the claims in this document go further than that scope, on purpose.
Prior analysis of this exact word = definition dataset found a directed
"definition graph" (word -> each word in its own definition that is itself an entry) with one giant strongly-connected component covering ~97% of nodes, and
showed via two null models (Erdos-Renyi, degree-preserving configuration model) that
this is largely a trivial consequence of graph density, not semantic signal. That
finding is the starting point here, not the goal — see
Graph construction for how it's reused (not
re-derived as a headline result) in this codebase.
data/ dataset loader/parser, LexiconSource adapter, text vocabulary
graph/ graph construction, descriptive stats, null models (sanity check)
training/ splits, negative/corruption sampling, metrics, training loops
models/ baselines, learned embeddings, directed GNN, corruption detector
experiments/ exp1_link_prediction/, exp2_corruption_detection/, generalization/
tests/ unit tests (pytest) -- run before trusting any result below
results/ JSON result exports (checked into results/*.json, not raw)
scripts/ CLI entrypoints (build_graph.py, run_exp1.py, run_exp2.py, ...)
PLAN.md the plan written before implementation, kept as-is
data/raw/knowledge-sample.txt: one word = definition entry per logical line,
plain text. data/adapter.py defines LexiconSource as the seam — swap in a
different wordlist in the same format by pointing PlainTextLexiconSource at a
different path; nothing downstream (graph/training/models) needs to change. A
different format (CSV, JSON, ...) needs a new LexiconSource subclass, not changes
to existing code.
Before writing the parser, the raw file was inspected directly (see PLAN.md).
Two real corruptions were found and are handled explicitly in data/loader.py,
logged (never silently fixed), and covered by unit tests in tests/test_loader.py:
- 5 lines have the word "means" replaced by a bare
=inside the definition text (e.g.instrument = a tool or = used to perform an action) — almost certainly an upstream generation bug (a" means "→" = "replace that also hit the word "means" itself, not just the field delimiter). Fixed by substituting "means" back in wherever a stray=appears inside definition text. - 1 line glues two records together with no newline between them
(
question's definition runs directly intoquestions = ...). Fixed using the already-known vocabulary (built from the unambiguous lines first) to find the correct split point, with a documented duplicate-key policy (first occurrence wins) sincequestionsalso has its own separate, differently-worded entry.
Both fixes are driven by structural signatures in the text, not hardcoded to these
specific words, so an unresolved = in a differently-corrupted dataset is logged as
an error and the raw text is kept (no data silently dropped) rather than crashing.
Result: 2707 unique entries, matching the node count from the prior research described above — see Graph construction for why the edge count differs somewhat from that prior work despite the same node count.
graph/build.py: directed edge word -> dep iff dep is a token in word's
definition and dep is itself a vocabulary entry (dep != word, multi-occurrences
collapsed to one edge) — exactly the definition given in the project brief, with no
additional stopword filtering.
Measured on this dataset: 2707 nodes, 12405 edges (mean out/in-degree 4.58), giant SCC = 2700 nodes (99.7%), 8 SCCs total, 0 isolated nodes.
This is higher than the previously-reported ~11061 edges / ~97% giant SCC on
"the same dataset". The likely reason, checked directly: the top in-degree nodes
here include not just the hub content words the prior research listed (act,
state, action, time, parts, having, multiple, place, process, ...)
but also a handful of very generic connector words that happen to be lexicon entries
themselves and are used as filler across huge numbers of definitions —
something (in-degree 511), from (154), one (72), not (64), through (59),
toward (59), between (57), without (57). Whether such connector words count as
"dependency edges" is exactly the kind of tokenization/scope choice that shifts edge
count without changing the graph's fundamental character. Rather than tune the
graph-building rule to match an old number, the literal definition from the project
brief was kept, and the sanity check was recomputed fresh on this exact graph
instead of assumed from the earlier run:
| giant SCC fraction | |
|---|---|
| real graph | 0.9974 |
| Erdos-Renyi (same n, m), 5 trials | 0.9794 ± 0.0026 |
| degree-preserving configuration model, 5 trials | 0.9971 ± 0.0010 |
Same qualitative conclusion as the prior work: giant-SCC size is barely
distinguishable from a random graph with the same density/degree sequence. This
number is used nowhere below as evidence of learned structure — it is reused
directly, unmodified, as the "randomized-graph" sanity-check null model in the
generalization tests (graph/null_models.py, shared by
experiments/generalization/randomized_graph.py).
A non-observed pair is documented throughout the code as "not written down as this
word's dependency in this dataset", never as "semantically impossible" — see
training/negatives.py's module docstring. All degree/structural statistics used to
build negatives are computed on the train graph only; a val/test edge's
degree-aware or hard-structural negatives are never informed by full-graph structure
that includes the very edges being predicted (training/negatives.py,
NegativeSampler._build_degree_buckets / _two_hop_candidates).
- easy — uniform random non-edge.
- degree_aware — a same-degree-bucket (train-graph degree, 10 buckets) node.
- hard_structural — a node within
u's 2-hop train-graph neighborhood (shares an intermediate node with a real path fromu) that isn't the true target. - lexical (optional, implemented) — a node drawn from the definitions of
u's train-graph neighbors (a second graph hop, still no external labels).
The hard_structural/lexical candidate pools are much smaller than easy/degree_aware ones for most words (2-hop neighborhoods and neighbor-definition vocabularies are small for typical, non-hub words):
| level | mean candidate pool size (target 49) |
|---|---|
| easy | 49.0 |
| degree_aware | 49.0 |
| hard_structural | 13.4 (median 11; 42% of queries have <10 candidates) |
| lexical | 12.9 (median 11; 41% of queries have <10 candidates) |
Hits@10 on hard_structural/lexical is measured against a much smaller field, so it is not directly comparable to Hits@10 on easy/degree_aware — e.g. the random baseline's Hits@10 is ~0.20 on easy/degree_aware but ~0.77 on hard_structural/lexical, purely because a random guess is much more likely to land in the top 10 when there are only ~11 candidates total. ROC-AUC, PR-AUC, and MRR remain informative across levels and are the metrics to trust for cross-level comparison; Hits@10 should only be compared within a level, against that level's own random-baseline row, not across levels. This is reported here rather than hidden because it changes how the result tables below should be read.
70/15/15 train/val/test split of the 12405 real edges (seeded, deterministic,
persisted to data/processed/splits/exp1_seed{N}.json) — chosen to leave a test set
large enough (~1860 edges) for stable ranking metrics while keeping most of the graph
for training. Models: random, degree_popularity, directed_common_neighbor
(directed adaptation: |successors(u) ∩ predecessors(v)|, i.e. length-2 directed
paths, since standard common-neighbors is undirected), embedding_dot_product (a
learned per-node embedding table with no message passing — 173k params, transductive
only), and directed_gnn (hand-rolled directed GraphSAGE-style network, 11.8k
params — see Why not PyTorch Geometric).
3 seeds (0, 1, 2), mean ± std, test set, ROC-AUC (pooled = all 4 levels combined):
| model | easy | degree_aware | hard_structural | lexical | pooled |
|---|---|---|---|---|---|
| random | 0.497±0.004 | 0.499±0.006 | 0.501±0.006 | 0.499±0.005 | 0.499±0.000 |
| degree_popularity | 0.709±0.007 | 0.555±0.003 | 0.554±0.012 | 0.448±0.012 | 0.604±0.006 |
| directed_common_neighbor | 0.567±0.002 | 0.565±0.002 | 0.303±0.005 | 0.288±0.002 | 0.508±0.002 |
| embedding_dot_product | 0.515±0.004 | 0.584±0.002 | 0.633±0.002 | 0.652±0.003 | 0.570±0.001 |
| directed_gnn | 0.717±0.016 | 0.564±0.007 | 0.610±0.024 | 0.489±0.023 | 0.621±0.014 |
MRR (same runs): random 0.09–0.28, degree_popularity 0.18–0.34, common_neighbor
0.15–0.17, embedding 0.17–0.46, directed_gnn 0.18–0.36 (full table with PR-AUC and
Hits@{1,5,10} per level in results/exp1_link_prediction/exp1_aggregate.json).
Honest reading of this table, not just the winning numbers:
- The GNN beats every baseline on
easyand on the pooled average, and clearly beatsdirected_common_neighboronhard_structural(0.610 vs 0.303) — the classic common-neighbor heuristic actively fails on hard negatives, because those negatives were specifically chosen to share neighbors with the true target, which is exactly the signal that heuristic relies on. - The GNN does not win everywhere. On
lexicalnegatives it is statistically indistinguishable from random (0.489), while the much simplerembedding_dot_productbaseline reaches 0.652 there and also beats the GNN onhard_structural(0.633 vs 0.610). A likely reason: the GNN's node features are purely structural (log-degree only, by design — see Node features), so it has no way to represent the specific local-vocabulary signal that lexical negatives are built from, while the embedding table can directly memorize per-node idiosyncrasies. This is reported as a real, unforced result, not smoothed into "the GNN wins overall". degree_popularityscores below random onlexical(0.448) — a popularity shortcut actively hurts once negatives are drawn from a plausible local context instead of the whole vocabulary.
Word-level 70/15/15 split (independent of Experiment 1's edge split, persisted to
data/processed/splits/exp2_seed{N}.json). CorruptionGenerator
(training/corruption.py) reuses NegativeSampler's degree-aware and
hard-structural logic for corruption levels 5 and 3 respectively, so "hard" means the
same thing in both experiments. Hard/degree corruption candidates for a val/test
word are computed on the subgraph induced by train words only
(training/splits.induced_subgraph_edges), so a held-out word's own connectivity
never informs which corruption candidates are considered plausible for it. Whole
definition swaps are also split-local: train targets borrow only train definitions,
validation targets only validation definitions, and test targets only test
definitions.
Model: CorruptionDetector (90.7k params) — word and definition tokens share one
embedding table trained from scratch (no pretrained vectors, see data/text_vocab.py);
the definition encoder is a mean-pool, not a recurrent/attention encoder, on
purpose (~1900 positive training examples per split is a real overfitting risk for a
higher-capacity sequence encoder, and the corruption levels being detected are about
which tokens appear, not their order).
An audit found that the original definition_swap implementation sampled borrowed
definitions from all 2,707 words, despite the word-level split. Consequently every
test definition had appeared as a negative training example—typically 66–86 times.
This directly contradicted its positive test label and caused the earlier systematic
below-chance result. The generator now requires a split-local definition pool, with
regression tests enforcing the boundary. All Experiment 2 controls, diagnostics,
encoder/objective ablations, aggregates, and figures below were regenerated from
scratch after the fix. The superseded pre-fix metrics are not used in any conclusion.
Corrected 3-seed result, mean ± std, test set (407 words):
| corruption level | ROC-AUC | PR-AUC |
|---|---|---|
| single_swap (1 token replaced) | 0.607 ± 0.009 | 0.595 ± 0.008 |
| multi_swap (all dependency tokens replaced) | 0.915 ± 0.010 | 0.928 ± 0.008 |
| neighbor_swap (topologically-close replacement) | 0.575 ± 0.009 | 0.565 ± 0.013 |
| definition_swap (whole definition borrowed within split) | 0.494 ± 0.010 | 0.506 ± 0.014 |
| structural_hard (degree-matched replacement) | 0.570 ± 0.006 | 0.555 ± 0.011 |
| pooled (all levels) | 0.632 ± 0.004 | 0.599 ± 0.002 |
Honest reading:
multi_swapis detected well (0.915): replacing all dependency tokens leaves a strong signal for a mean-pooled representation to pick up.single_swap,neighbor_swap, andstructural_hardare modest (0.57–0.61): consistent, above-chance, but far from strong. A single token swap is diluted by averaging over the rest of a definition.- Correcting the leakage moves
definition_swapfrom the invalid pre-fix0.434 ± 0.009to0.494 ± 0.010. The inversion is gone, but the detector still does not learn held-out word/definition compatibility. That remaining chance-level result is now the valid scientific finding.
A read-only follow-up replays the corrected test corruptions and verifies them against
the saved metrics (scripts/run_definition_swap_diagnosis.py):
| seed | detector ROC-AUC | scalar-only baseline ROC-AUC | correct-pair score delta > 0 |
|---|---|---|---|
| 0 | 0.505 | 0.510 | 49.9% |
| 1 | 0.481 | 0.468 | 46.9% |
| 2 | 0.495 | 0.522 | 47.9% |
Both the detector and the standalone scalar baseline are now around chance, and the detector prefers the correct member of a pair about half the time. Full per-seed distributions, correlations, token-level details, and coefficients are exported for seed 0, seed 1, seed 2, and the compact per-seed summary.
A logistic regression on six borrowed-minus-own scalar deltas can still reproduce the detector's preference, but no longer predicts correctness:
| seed | preference ROC-AUC | preference accuracy | correctness ROC-AUC |
|---|---|---|---|
| 0 | 0.969 | 0.897 | 0.499 |
| 1 | 0.999 | 0.980 | 0.471 |
| 2 | 1.000 | 0.990 | 0.480 |
This is a model-behavior diagnostic, not evidence of a successful shortcut: scalar
geometry explains which arbitrary choice the chance-level detector makes, but that
choice has no useful relation to the correct definition. Inference-only L2
normalization also does not help (0.505→0.489, 0.481→0.470, 0.495→0.477).
The next experiment separates encoder choice from training objective. A0 is the
existing 32-dimensional mean-pool; A1 is a single-layer bidirectional GRU with 16
hidden units per direction, using its concatenated final states (32 dimensions).
B0 is the existing independent BCE objective; B1 is the untuned logistic pairwise
loss softplus(-(score_own - score_corrupted)). A0×B0 reuses the existing runs;
the other three cells were trained from scratch on the same saved splits, corruption
streams, hyperparameters, and seeds. No pretrained representation or test-driven
tuning was introduced.
Primary result below is ROC-AUC mean±std / PR-AUC mean±std over seeds 0/1/2:
| cell | single | multi | neighbor | definition | structural | pooled |
|---|---|---|---|---|---|---|
| A0×B0 mean+BCE | 0.607±0.009 / 0.595±0.008 | 0.915±0.010 / 0.928±0.008 | 0.575±0.009 / 0.565±0.013 | 0.494±0.010 / 0.506±0.014 | 0.570±0.006 / 0.555±0.011 | 0.632±0.004 / 0.599±0.002 |
| A0×B1 mean+pairwise | 0.599±0.007 / 0.588±0.005 | 0.910±0.019 / 0.922±0.018 | 0.567±0.007 / 0.561±0.009 | 0.494±0.006 / 0.503±0.009 | 0.569±0.002 / 0.563±0.008 | 0.628±0.004 / 0.598±0.002 |
| A1×B0 BiGRU+BCE | 0.601±0.010 / 0.582±0.003 | 0.879±0.010 / 0.888±0.013 | 0.574±0.002 / 0.567±0.007 | 0.503±0.003 / 0.503±0.002 | 0.573±0.013 / 0.562±0.018 | 0.626±0.004 / 0.594±0.003 |
| A1×B1 BiGRU+pairwise | 0.597±0.010 / 0.578±0.019 | 0.891±0.014 / 0.897±0.019 | 0.574±0.013 / 0.565±0.011 | 0.503±0.008 / 0.503±0.006 | 0.571±0.004 / 0.560±0.008 | 0.627±0.004 / 0.594±0.009 |
The corrected ablation is consistent across seeds: mean+BCE remains best overall,
while both BiGRU cells reach only 0.503 on definition_swap, a negligible change
from chance. Pairwise training does not improve the task. Order preservation and the
alternative objective therefore do not solve the remaining lack of transferable
word/definition matching. Parameter count rises only from 90,689 to 95,489, so this
remains a small encoder comparison rather than a capacity contest. Per-seed training
histories, metadata, full metrics, and paired-control coefficients are exported under
the encoder-ablation results directory.
The left panel shows performance per corruption type (the dashed line is chance),
the middle panel isolates the remaining definition_swap performance gap, and the
right panel shows how strongly scalar deltas predict the detector's preference—not
whether that preference is correct. Regenerate both figures with
python scripts/plot_encoder_ablation.py. The reruns contain complete histories.
Experiment 2 holds target words out entirely. Because the model represents a target as an embedding-table row, many held-out targets receive little or no semantic training signal: about 15–16% never occur in a train definition, and roughly half occur at most once. Experiment 3 separates model capability from this information bottleneck in two ways.
First, a deliberately transductive known-word ceiling trains on all 2,707 real word/definition entries, then evaluates independent validation and test corruption streams. This is a valid closed-lexicon consistency test, but not an unseen-word generalization claim: the positive dictionary entries are known and may be memorized.
| corruption level | unseen target words (Exp. 2) | known target words (Exp. 3) |
|---|---|---|
| single_swap | 0.607 ± 0.009 | 0.884 ± 0.005 |
| multi_swap | 0.915 ± 0.010 | 0.986 ± 0.002 |
| neighbor_swap | 0.575 ± 0.009 | 0.895 ± 0.005 |
| definition_swap | 0.494 ± 0.010 | 0.933 ± 0.002 |
| structural_hard | 0.570 ± 0.006 | 0.849 ± 0.005 |
| pooled | 0.632 ± 0.004 | 0.909 ± 0.003 |
The same 90,689-parameter mean+BCE model therefore can learn word/definition
compatibility when its target-word rows are informed. The corrected chance-level
inductive result is not primarily a capacity or optimization failure. All three
known-word curves were still improving when the predeclared 300-epoch ceiling was
reached, so 0.933 is a conservative measured ceiling rather than evidence that the
model has exhausted this easier protocol.
Second, the corrected Experiment-2 checkpoints are evaluated by how often each held-out target word occurs inside train definitions:
| train-definition mentions | mean words/seed | definition ROC-AUC | pairwise accuracy |
|---|---|---|---|
| 0 | 63 | 0.459 ± 0.018 | 0.483 ± 0.039 |
| 1 | 148 | 0.467 ± 0.014 | 0.448 ± 0.011 |
| 2–5 | 146 | 0.500 ± 0.026 | 0.479 ± 0.023 |
| 6+ | 50 | 0.578 ± 0.049 | 0.594 ± 0.054 |
This exposure association supports the information-bottleneck hypothesis, though it is observational: frequently mentioned words may also be easier or more central. The next honest inductive model should construct the target representation from train-only incoming definition contexts (with a character/subword fallback for zero- exposure words), then train an explicit word-to-definition retrieval objective.
Full seeded outputs and histories are under
results/exp3_known_word_consistency/.
The first inductive follow-up removes the atomic target-ID dependency. Held-out words are encoded from their characters and, when enabled, up to eight definitions of training words that mention the target. A target's own definition is never a context source, val/test definitions are excluded, and the target token itself is removed from each context. Across seeds, 2,318–2,339 of 2,707 words have at least one such context; all others use the character fallback.
Two heads are tested: the existing flexible classifier and a cosine head that can score only direct geometric agreement between target and definition representations.
| target encoder / head | definition ROC-AUC | pooled ROC-AUC |
|---|---|---|
| ID-only inductive control | 0.494 ± 0.010 | 0.632 ± 0.004 |
| character / classifier | 0.499 ± 0.009 | 0.633 ± 0.005 |
| context+character / classifier | 0.499 ± 0.006 | 0.631 ± 0.004 |
| character / cosine | 0.522 ± 0.035 | 0.626 ± 0.004 |
| context+character / cosine | 0.492 ± 0.007 | 0.623 ± 0.002 |
| known-word ceiling (Exp. 3) | 0.933 ± 0.002 | 0.909 ± 0.003 |
The character-cosine mean is driven by one seed (0.489/0.570/0.506) and is not a
stable improvement. More importantly, adding the flat incoming-text context helps
neither head. This rejects the simplest context hypothesis: concatenating referring
definitions into a mean-pooled bag loses source identity, edge direction, and which
tokens co-occurred in which definition. It does not mean the closed-lexicon graph is
uninformative; Experiment 3's exposure trend remains, but exploiting it requires an
encoder that preserves relational structure—e.g. message passing over referring
word nodes or attention over separate context definitions—plus an explicit retrieval
objective.
Full histories and metrics are in
results/exp4_context_encoder/.
Experiment 5 preserves each lexicon reference as a directed edge. Message-passing
edges may originate only from train-word definitions; held-out definitions cannot
send messages or reveal their contents. A two-layer directed GraphSAGE target encoder
is compared under ordinary BCE, pairwise cosine matching, and full in-split InfoNCE.
The latter ranks each train word's own definition against every other train definition
on every update. Evaluation includes the corrected corruption suite and 50-way held-
out definition retrieval (random expectation: MRR about 0.09, Hits@1 0.02).
| graph objective | definition ROC-AUC | pooled ROC-AUC | retrieval MRR | Hits@1 |
|---|---|---|---|---|
| BCE | 0.495 ± 0.010 | 0.631 ± 0.002 | 0.087 ± 0.004 | 0.014 ± 0.005 |
| pairwise cosine | 0.491 ± 0.007 | 0.625 ± 0.003 | 0.085 ± 0.001 | 0.013 ± 0.004 |
| full InfoNCE | 0.523 ± 0.015 | 0.512 ± 0.014 | 0.099 ± 0.009 | 0.022 ± 0.009 |
| BCE + 0.05 InfoNCE | 0.492 ± 0.006 | 0.623 ± 0.001 | 0.086 ± 0.002 | 0.016 ± 0.003 |
| BCE + 0.10 InfoNCE | 0.492 ± 0.005 | 0.621 ± 0.002 | 0.088 ± 0.003 | 0.020 ± 0.005 |
BCE and pairwise training reproduce the earlier chance-level matching result despite
preserving graph direction. Full InfoNCE is the first graph variant to move both
matching measurements in the intended direction: definition ROC-AUC is
0.502/0.530/0.536 and retrieval MRR is 0.090/0.098/0.110 across seeds. The gain is
small and retrieval remains close to random, so this is evidence of a weak transferable
signal—not a solved inductive task. InfoNCE also sacrifices the non-relational
corruption signals because it is trained only for matching.
A predeclared multitask follow-up combines BCE with InfoNCE weights 0.05 and 0.10.
Checkpoint selection averages validation corruption ROC-AUC with validation retrieval
pair-accuracy, so neither task can dominate selection. Both variants retain much of
BCE's pooled performance but lose InfoNCE's small matching gain. The objectives do
not share this single representation constructively under simple weighted addition;
a larger weight would mostly approach the already measured pure-InfoNCE endpoint.
Each test word has only about three incoming references from train definitions on average, and 60–65 of 407 have none. With this 2,707-entry sample, two message-passing layers therefore have little evidence to aggregate. Scaling the lexicon should be tested by controlled data-size curves before increasing model capacity: if incoming coverage and retrieval rise together, the larger 400k–500k lexicon is exactly the missing resource; if not, the next change should be definition-level attention or a joint multitask BCE+InfoNCE objective.
Full seeded metrics are under
results/exp5_graph_retrieval/.
The pure-InfoNCE hypothesis is stress-tested in three ways. First, retrieval expands from 50 candidates to all 407 held-out definitions. Second, real definitions owned by directly connected graph words receive 2× denominator weight during InfoNCE. Third, the saved joint-λ=.10 checkpoints are used to measure cosine similarity between BCE and InfoNCE gradients on the same 512 train words.
| model | definition ROC-AUC | 50-way MRR | full 407-way MRR | full Hits@1 |
|---|---|---|---|---|
| pure InfoNCE | 0.523 ± 0.015 | 0.099 ± 0.009 | 0.0199 ± 0.0045 | 0.0049 ± 0.0035 |
| graph-hard InfoNCE | 0.512 ± 0.009 | 0.092 ± 0.002 | 0.0169 ± 0.0007 | 0.0025 ± 0.0000 |
| random expectation | 0.500 | ≈0.090 | ≈0.0162 | ≈0.0025 |
Pure InfoNCE's small signal survives the harder full candidate set, but remains weak and seed-dependent; hard-negative reweighting reduces it. Direct graph neighbors are not automatically useful semantic confounders in this sparse sample, so emphasizing them adds difficulty faster than it adds learnable evidence.
The gradient diagnosis also refines the earlier “BCE pulls in the opposite direction” interpretation:
| parameter group | mean BCE/InfoNCE gradient cosine |
|---|---|
| all trainable parameters | +0.552 |
| character + word encoder | +0.542 |
| directed graph layers | +0.194 |
| definition embedding | +0.476 |
The objectives are not globally antagonistic. Their graph-layer gradients are only
weakly aligned (0.069/0.471/0.041 per seed), while BCE supplies an easier and much
denser optimization signal. The retrieval feature is therefore more plausibly
under-supported or dominated than actively reversed. This shifts the next decision
toward controlled graph-coverage scaling and separate retrieval capacity, not more
loss-weight tuning.
Full outputs are under
results/exp6_hard_retrieval/.
Pure graph InfoNCE is retrained on nested prefixes containing 25%, 50%, 75%, and 100% of each seed's train definitions. Validation/test words, architecture, hyperparameters, and candidate sets remain fixed. Each larger point contains every definition from the smaller point plus additional ones, so this measures the combined effect of more training pairs and more visible reference edges without changing the evaluation population.
| train fraction | train definitions | visible edges | zero-context test words | mean incoming refs | definition AUC | 50-way MRR | 407-way MRR |
|---|---|---|---|---|---|---|---|
| 25% | 473 | 2,164 | 60.3% | 0.81 | 0.500 ± 0.022 | 0.092 ± 0.004 | 0.0178 ± 0.0015 |
| 50% | 947 | 4,351 | 41.0% | 1.60 | 0.516 ± 0.005 | 0.100 ± 0.007 | 0.0170 ± 0.0004 |
| 75% | 1,420 | 6,530 | 26.7% | 2.43 | 0.513 ± 0.028 | 0.093 ± 0.008 | 0.0166 ± 0.0025 |
| 100% | 1,894 | 8,672 | 15.6% | 3.21 | 0.511 ± 0.003 | 0.098 ± 0.006 | 0.0168 ± 0.0018 |
Coverage improves strongly and monotonically, but retrieval does not. Across all 12
runs, Spearman correlation between mean incoming references and definition ROC-AUC
is ρ=0.07 (p=0.83), with 50-way MRR ρ=0.19 (p=0.56) and full MRR ρ=-0.40
(p=0.20). None supports density as the controlling variable under this encoder.
This rejects the simple extrapolation that a much larger lexicon will succeed merely because it supplies more edges. A 400k–500k resource may still help through richer language and repeated semantic evidence, but the present mean-message GraphSAGE cannot exploit added coverage predictably. The next architectural step should retain each referring definition as a separate evidence item and select among them with definition-level attention; raw neighbor means discard too much content and source identity.
Full seeded curves are under
results/exp7_density_curve/.
To test the remaining closed-lexicon hypothesis, up to eight train definitions that mention a target are retained as separate evidence items. Each is encoded separately; character-derived target queries attend over evidence keys/values before pure InfoNCE retrieval. Target tokens are removed from evidence, target-own and val/test definitions are unavailable, and zero-evidence targets use the character fallback.
The matched character-only InfoNCE control produces a much larger result than any previous inductive model:
| target representation | definition ROC-AUC | 50-way MRR | 407-way MRR | full Hits@1 |
|---|---|---|---|---|
| shuffled spelling control | 0.507 ± 0.008 | 0.095 ± 0.003 | 0.0167 ± 0.0009 | 0.0016 ± 0.0012 |
| real spelling | 0.671 ± 0.007 | 0.236 ± 0.011 | 0.0830 ± 0.0029 | 0.0418 ± 0.0040 |
| spelling + evidence attention | 0.664 ± 0.006 | 0.192 ± 0.001 | 0.0532 ± 0.0011 | 0.0172 ± 0.0053 |
| 407-way random expectation | 0.500 | — | ≈0.0162 | ≈0.0025 |
The shuffled control permutes the exact same spelling inventory across word IDs and returns every metric to chance. Only one of 2,707 definitions contains its own target word literally, so direct answer-token leakage cannot explain the result. Full-batch InfoNCE has learned transferable orthographic/morphological regularities: related word forms share recurring definitional language, allowing unseen target strings to align with plausible definitions.
Separate evidence attention does not improve this strong control and consistently reduces retrieval. Incoming mentions are often usage/context relations rather than clean definitions; attending over them from a spelling query introduces noisy evidence and may overwrite a useful morphological representation. This rejects the first evidence-fusion design, not the broader closed-lexicon premise.
This is the strongest inductive result in the project so far and materially supports the hypothesis that a closed lexicon can teach reusable structure without pretrained language representations. The next model should preserve the character-InfoNCE path as the primary encoder and add evidence only through a gated residual whose default is exactly zero, so validation must demonstrate incremental value before context can alter the proven spelling representation.
Full results are under
results/exp8_evidence_attention/.
The strongest character result is retested with complete orthographic families held
together. A fixed, external-data-free rule normalizes common inflections and suffixes
(-s/-es, -ed/-ing, -ly, -ness, -ment, -ity, -tion, etc.); all words with
the same resulting key enter the same train, validation, or test partition. The 2,707
words form 1,902 conservative groups, and every saved seed has exactly zero family-key
overlap among splits. Because this heuristic intentionally avoids aggressive stemming,
it is stricter than a random word split but not a claim that every linguistic family
is perfectly identified.
| protocol | definition ROC-AUC | 50-way MRR | 407-way MRR | full Hits@1 |
|---|---|---|---|---|
| random word split, real spelling (Exp. 8) | 0.671 ± 0.007 | 0.236 ± 0.011 | 0.0830 ± 0.0029 | 0.0418 ± 0.0040 |
| disjoint families, real spelling | 0.630 ± 0.009 | 0.185 ± 0.007 | 0.0529 ± 0.0064 | 0.0188 ± 0.0046 |
| disjoint families, shuffled spelling | 0.505 ± 0.013 | 0.085 ± 0.003 | 0.0152 ± 0.0005 | 0.0025 ± 0.0000 |
Holding families out removes a meaningful portion of the gain, confirming that close
inflectional/derivational relatives helped the random split. It does not erase the
signal: real spelling remains far above its shuffled control on every metric and is
stable across definition-AUC seeds (0.638/0.618/0.635). The model therefore learns
orthographic-to-definitional regularities that generalize beyond the conservative
families captured by the rule, though this still need not equal broad semantic
understanding. Saved qualitative rankings illustrate both sides: down retrieves its
definition at rank 1, while many sampled words remain poorly ranked and some high
scorers are not obviously semantically related.
Full metrics, auditable family splits, and top-five retrieval examples are under
results/exp9_morphology_audit/.
The three real-spelling checkpoints from Experiment 9 are tested on 100 synthetic
nonce cases: 20 unseen stems combined with five English affixes (un-, -less,
-ness, -er, and -ly). Each form ranks five short affix-meaning templates. This
is an exploratory forced-choice probe rather than a new supervised task: the models
are frozen, the nonce words never enter training, and the templates use only the
existing lexicon vocabulary.
| nonce spelling | 5-way meaning accuracy | MRR |
|---|---|---|
| canonical affix position | 0.487 ± 0.019 | 0.673 ± 0.020 |
| affix moved to wrong side | 0.487 ± 0.019 | 0.673 ± 0.020 |
| all characters shuffled | 0.487 ± 0.019 | 0.673 ± 0.020 |
| random expectation | 0.200 | ≈0.457 |
The apparent above-chance nonce signal is not evidence of productive affix
rules. The current character encoder mean-pools individual character embeddings, so
all permutations of the same multiset of letters are architecturally indistinguishable.
The audit confirms this directly: canonical, moved, and shuffled representations are
equal for 100% of cases within 1e-6 (maximum numerical difference <1.8e-7) for
all seeds, and therefore produce identical rankings. Experiment 9's surviving signal
must presently be described as order-free character-composition correlation, not as
learned prefix/suffix syntax.
This gives a concrete architectural next step: replace the character mean with an order-sensitive encoder (character CNN, BiGRU, or positional character n-grams), retrain under the same family split, and repeat this exact audit. A genuine productive effect must beat chance for canonical forms while dropping for moved and shuffled controls.
Full per-seed outputs are under
results/exp10_productive_morphology/.
The character mean is replaced by a small boundary-aware CNN over character bigrams and trigrams. Everything else remains matched: 32-dimensional representations, pure InfoNCE, the exact Experiment-9 family split for each seed, training schedule, and from-scratch lexicon-only supervision. The CNN has 92,225 trainable parameters versus 94,881 in the Experiment-9 implementation (which retains unused evidence-attention modules), so the improvement is not explained by a parameter-count increase.
| family-split encoder | definition ROC-AUC | 50-way MRR | 407-way MRR | full Hits@1 |
|---|---|---|---|---|
| character mean (Exp. 9) | 0.630 ± 0.009 | 0.185 ± 0.007 | 0.0529 ± 0.0064 | 0.0188 ± 0.0046 |
| character CNN | 0.660 ± 0.005 | 0.195 ± 0.009 | 0.0618 ± 0.0057 | 0.0254 ± 0.0050 |
The same frozen-CNN nonce audit now distinguishes order:
| nonce spelling | 5-way meaning accuracy | MRR |
|---|---|---|
| canonical affix position | 0.480 ± 0.041 | 0.679 ± 0.022 |
| affix moved to wrong side | 0.340 ± 0.051 | 0.564 ± 0.043 |
| all characters shuffled | 0.393 ± 0.046 | 0.601 ± 0.018 |
| random expectation | 0.200 | ≈0.457 |
Unlike the mean encoder's mathematically identical outputs, the CNN gives canonical
forms higher accuracy than both controls in every seed. The gains are not merely a
probe artifact: held-out-family AUC rises by 0.030 and full MRR by about 17% relative.
This is positive evidence that local character order and word boundaries carry a
transferable signal in this lexicon. It is still not proof of unrestricted semantic
composition: moved and shuffled forms remain above chance, showing that character
inventory remains predictive, and the five hand-written meaning templates are a
small diagnostic. The supported claim is narrower: fixing order blindness improves
real held-out retrieval and enables a reproducible canonical-affix preference.
Full per-seed histories and audit outputs are under
results/exp11_character_cnn/.
The CNN is trained with one coherent bidirectional contrastive objective. In the original direction, every real training spelling retrieves its real definition. In the reverse direction, every definition must retrieve its real spelling over all other real spellings plus deterministic same-letter shuffles and reversals. Only training-family words generate counterfactuals; validation/test families and nonce forms remain unseen. No synthetic meaning labels, external data, or pretrained model is introduced. Checkpoint selection still uses ordinary real validation retrieval, not the order probe.
| family-split model | definition ROC-AUC | 50-way MRR | 407-way MRR | full Hits@1 |
|---|---|---|---|---|
| CNN InfoNCE (Exp. 11) | 0.660 ± 0.005 | 0.195 ± 0.009 | 0.0618 ± 0.0057 | 0.0254 ± 0.0050 |
| CNN + order contrast | 0.674 ± 0.010 | 0.223 ± 0.006 | 0.0617 ± 0.0070 | 0.0188 ± 0.0076 |
The primary matched control uses each of the 407 real held-out definitions to choose between its correct unseen spelling and a same-letter corruption:
| correct-spelling pair accuracy | CNN InfoNCE | + order contrast |
|---|---|---|
| versus shuffled | 0.618 ± 0.024 | 0.710 ± 0.005 |
| versus reversed | 0.637 ± 0.040 | 0.733 ± 0.019 |
This increase occurs in every seed and is accompanied by improved definition AUC and 50-way retrieval; full MRR is preserved, although Hits@1 falls. It establishes that order sensitivity is transferable beyond training families and that matched same-letter contrast strengthens it rather than merely fitting the nonce templates.
The stricter productive-morphology claim remains mixed. Frozen-model nonce accuracy
is 0.493 ± 0.029 canonical, 0.367 ± 0.103 with the affix moved, and
0.280 ± 0.051 shuffled (chance 0.200). Canonical beats shuffled in every seed,
but does not beat the moved-affix control in every seed. Experiment 12 therefore gives
a firm positive answer for learned character order, but not yet a definitive claim
that the model consistently assigns compositional meaning to affix position.
Full outputs are under
results/exp12_order_contrast/.
This audit removes hand-written nonce meanings from the primary claim. Conservative
real base/derived pairs are extracted directly from the lexicon (accurate→accurately,
accept→accepting, connect→connected, etc.). For each eligible affix, a semantic
prototype is computed solely from the mean training-definition difference between
derived and base entries. Frozen CNN spelling differences from entirely held-out
base/derived pairs must then identify the matching semantic prototype. Any pair whose
base and derivative cross a saved split boundary is excluded, never repaired or moved.
The small lexicon supports -ly, -ing, and -ed in all seeds and -ness in one;
this yields 28/43/39 test pairs. un- and -less do not meet the predeclared minimum,
so no result for them is claimed. Chance varies with three or four eligible affixes
(0.333/0.250/0.333).
| frozen encoder / test delta | affix-prototype accuracy | MRR |
|---|---|---|
| Exp. 12 definition delta (semantic oracle) | 0.641 ± 0.027 | 0.793 ± 0.012 |
| Exp. 12 canonical spelling delta | 1.000 ± 0.000 | 1.000 ± 0.000 |
| same affix moved to wrong side | 0.724 ± 0.139 | 0.846 ± 0.079 |
| derived spelling shuffled | 0.488 ± 0.091 | 0.701 ± 0.055 |
| canonical with cyclically wrong prototype labels | 0.000 ± 0.000 | 0.398 ± 0.038 |
The direct paired test is stronger than comparing aggregate classification rates:
the canonical spelling receives a higher correct-semantic-prototype score than the
moved form for 0.969 ± 0.044 of held-out pairs and than its same-letter shuffle for
1.000 ± 0.000. Both preferences are positive in every seed. The ordinary Exp.-11
CNN already shows a signal (0.833 moved, 0.992 shuffled), while matched order
training strengthens it substantially.
The moved control remaining well above chance (0.724) is informative rather than a
failed control. Affix identity is still recoverable from its characters and local
n-grams even at the wrong boundary; that order-free/partly local channel was already
visible in Experiments 10–12. The matched comparison isolates the additional positional
component: with affix characters held constant, the canonical placement receives the
higher correct-semantic-prototype score for 96.9% of pairs. The model therefore uses
both which character pattern is present and where that pattern occurs.
Taken together with the wrong-label collapse, this supports the project's central information hypothesis in a precise form: definitions are not merely graph wiring or targets for an InfoNCE trick. Their repeated wording supplies a semantic coordinate system in which a spelling transformation learned on some families predicts a definition transformation on unseen families. InfoNCE supplies the learning pressure; the reusable correspondence being recovered comes from regularities inside the definitions themselves.
Within the measured suffixes, this closes the narrow hypothesis: the model has learned a train-definition-derived, transferable association between affix position and a consistent change in definition space, rather than only recognizing the character inventory. It does not establish the same result for scarce prefixes/suffixes, irregular morphology, arbitrary new concepts, or general language understanding. The perfect classification also reflects an easy three/four-affix discrimination task; the paired controls and wrong-label falsification support the effect, while the reported scope prevents turning it into a broader claim than the data permits.
Full per-seed pair counts, exclusions, examples, and metrics are under
results/exp13_semantic_affix/.
The final morphology test asks for a consequence, not an affix label. For every held-out real base/derived pair, the query is constructed without its derived definition:
encoded base definition + mean train-only affix definition delta.
That query must retrieve the hidden derived definition among all 407 definitions in the family-held-out test split. The correct affix is compared with the unchanged base, a cyclically wrong affix, and the model's direct derived-spelling query. The latter is a native cross-modal reference, not an oracle or guaranteed upper bound. Affix eligibility, pair counts, and cross-split exclusions are identical to Experiment 13.
| Exp.-12 representation query | 407-way MRR | Hits@5 | median rank | mean rank |
|---|---|---|---|---|
| base definition only | 0.116 ± 0.016 | 0.254 ± 0.049 | 35.0 | 94.0 |
| base + correct train affix delta | 0.159 ± 0.021 | 0.359 ± 0.052 | 16.3 | 63.8 |
| base + wrong affix delta | 0.070 ± 0.015 | 0.141 ± 0.035 | 68.7 | 118.7 |
| derived spelling query | 0.075 ± 0.014 | 0.124 ± 0.027 | 86.7 | 128.3 |
| random MRR expectation | ≈0.016 | — | ≈204 | ≈204 |
On the matched per-pair comparison, correct composition gives the true derived
definition a higher score than base-only for 0.716 ± 0.052 of pairs and a better
rank for 0.623 ± 0.048; versus the wrong affix these become 0.806 ± 0.031 and
0.762 ± 0.044. The same qualitative result also holds for the ordinary Exp.-11 CNN
(MRR 0.103→0.158 for base-only→correct delta), so it is not created solely by the
order-contrastive objective.
This is positive evidence for limited compositional/analogical inference: a semantic
operation estimated from training families improves prediction of a hidden definition
in unseen families, and the operation's identity matters. It is not full reconstruction
(MRR 0.159, not 1.0), and lexical similarity in the base definition already provides
a strong starting point. The justified claim is therefore that the learned affix
operation contributes reusable semantic information beyond that base—not that the
model can generate arbitrary definitions or perform unrestricted reasoning.
Full per-seed ranks, comparisons, and examples are under
results/exp14_compositional_reconstruction/.
The strongest feasible causal intervention keeps the complete spelling inventory, definition-text inventory, family splits, CNN, bidirectional order-contrastive objective, parameter count, and training schedule fixed. In the counterfactual world, only train/validation word-to-definition assignments are independently deranged within their split: every definition is still used exactly once, but no word retains its own. Test definitions remain real and untouched. Each seed therefore changes only whether the training definitions are meaningfully attached to their spellings.
| held-out metric | normal definitions | permuted assignments |
|---|---|---|
| 407-way retrieval MRR | 0.0617 ± 0.0070 | 0.0174 ± 0.0043 |
| full Hits@1 | 0.0188 ± 0.0076 | 0.0016 ± 0.0023 |
| affix-prototype accuracy | 1.000 ± 0.000 | 0.332 ± 0.128 |
| canonical-over-moved preference | 0.969 ± 0.044 | 0.628 ± 0.061 |
| compositional reconstruction MRR | 0.159 ± 0.021 | 0.083 ± 0.005 |
| correct-over-wrong-affix score preference | 0.806 ± 0.022 | 0.570 ± 0.064 |
The intervention sends ordinary full retrieval essentially to its random expectation
(≈0.0162) and affix classification to the varying three/four-class chance region.
More specifically, adding the correct affix delta improves reconstruction over
base-only by +0.0429 MRR in the real world but only +0.0054 after permutation.
Every reported degradation has the same sign in all three seeds.
This supplies the missing causal control for the central information hypothesis. The model cannot recover the effect from the same vocabulary, character patterns, definition style, model architecture, or InfoNCE mechanics once meaningful assignment is removed. The reusable morphological and compositional signal therefore depends on which definitions belong to which words. Definitions are not merely passive candidate texts: their structured attachment is causally necessary for the measured transfer.
The intervention does not show human-like understanding or prove that every token in a definition is used semantically. It does rule out the broad alternative that these results arise from spelling statistics and contrastive machinery alone. Only five double-affix chains exist in the 2,707-entry lexicon, so the proposed two-operation Test 15B is explicitly deferred rather than reported from an underpowered sample.
Full seeded training histories, derangement checks, and downstream audits are under
results/exp15_definition_intervention/.
The natural 2,707-entry lexicon contains only five usable two-affix chains, so it cannot support a powered multi-step claim. Rather than over-report those five cases, Experiments 16–19 use a separate, explicitly synthetic factorial microlexicon. Sixty nonce bases each have a unique concept token and three suffix operations with explicit semantic tokens. All primitives and selected two-operation combinations are trained; other pairs and all three-operation chains are held out. This tests architectural systematicity under perfect coverage, not natural-English understanding.
Experiment 16 uses order-sensitive encoders on both spelling and definition sides. An initial mean-definition run was rejected because it made reversed operations mathematically identical; only the corrected BiGRU result is reported:
| held-out composition | candidates | MRR | Hits@1 | Hits@5 |
|---|---|---|---|---|
| unseen two-operation combinations | 72 | 0.978 ± 0.016 | 0.958 ± 0.030 | 1.000 |
| unseen longer three-operation chains | 360 | 0.546 ± 0.057 | 0.386 ± 0.067 | 0.746 ± 0.050 |
Experiment 17 removes the candidate list with an autoregressive GRU decoder. No test definition can be copied exactly because its concept/operation sequence is held out; triple targets are also longer than every primitive training target.
| freely generated definitions | exact match | all-token accuracy | non-concept sequence accuracy |
|---|---|---|---|
| unseen doubles | 0.847 ± 0.157 | 0.947 ± 0.056 | 0.929 ± 0.074 |
| length-extrapolated triples | 0.070 ± 0.051 | 0.616 ± 0.066 | 0.521 ± 0.082 |
The model can freely express most new two-step combinations, but reliable three-step generation fails: it often stops early, repeats, or swaps an operation. Retrieval therefore substantially overstates its ability to produce a complete reasoning chain.
Experiment 18 varies synthetic base concepts (15/30/60/120, corresponding to
114/228/456/912 training entries). Triple MRR is 0.549/0.590/0.546/0.667.
The largest condition is best, but the curve is non-monotonic and seed variance is
large. This is evidence that the controlled task can benefit from coverage, not a
scaling law and not evidence about 400k natural definitions. A genuine large-lexicon
scale test remains blocked on acquiring and documenting such a dataset.
Experiment 19 evaluates three matched binary skills on all 360 held-out triples:
| controlled task | accuracy | chance |
|---|---|---|
| correct versus reversed operation order | 0.835 ± 0.104 | 0.500 |
| complete chain versus last operation removed | 0.853 ± 0.028 | 0.500 |
| correct versus wrong base concept | 0.988 ± 0.017 | 0.500 |
Together these tests support systematic multi-step discrimination and strong unseen- combination retrieval in a controlled language. They also locate a sharp limitation: exact free generation at an unseen greater depth is only 7%. This is not general language understanding; it is a reproducible demonstration that the architecture can bind a base, preserve operation order, and combine learned primitives, while its decoder does not yet robustly execute longer chains.
Full outputs are under
Experiment 16,
Experiment 17,
Experiment 18, and
Experiment 19.
Before importing a larger lexicon, scripts/audit_lexicon.py now creates a canonical
JSONL record layer with content-stable IDs, source provenance, normalized fields,
tokens, and explicit quality flags. It also audits exact duplicate definitions and
uses deterministic MinHash/LSH candidate generation followed by exact shingle-Jaccard
verification for near duplicates. Original source files remain immutable; suspicious
records are flagged rather than silently discarded.
Leakage groups are the union of conservative suffix families, known-root prefix
relations (un-/re-/dis-/non- when the root exists), exact definitions, and verified
near-duplicate definitions. Entire connected groups enter one strict split. On the
current lexicon the audit reports:
| audit item | result |
|---|---|
| canonical / unique-word records | 2,707 / 2,707 |
| very short definitions | 4 |
| definitions containing their headword | 1 |
| exact duplicate-definition groups | 4 |
| verified near-duplicate pairs | 4 |
| strict leakage groups / largest group | 1,860 / 10 |
| words participating in measured affix pairs | 552 |
| real two-affix chains | 5 |
| definition tokens, mean / range | 7.14 / 2–17 |
All three generated strict splits contain 1,895/406/406 records and zero group
overlap by construction. Outputs are canonical JSONL,
the audit report, and stable-ID split manifests
under data/processed/audit/.
The near-duplicate stage is deliberately auditable and dependency-free. Its SQLite- backed MinHash/LSH index uses token unigrams and ordered bigrams, cardinality-bounded candidate queries, and exact Jaccard verification. No large bucket is silently skipped.
The open decoder question is now separated experimentally (Experiment 20). The
original short-chain teacher-forced model is compared with low-rate scheduled sampling
and with ordinary teacher forcing after adding length-matched triple examples. All
variants are tested on held-out triple combinations:
| decoder training | free exact | free token acc. | teacher-forced exact | teacher-forced token acc. |
|---|---|---|---|---|
| short targets, teacher forcing | 0.070 ± 0.051 | 0.616 ± 0.066 | 0.070 ± 0.051 | 0.729 ± 0.066 |
| short targets, 10% scheduled sampling | 0.092 ± 0.114 | 0.681 ± 0.069 | 0.092 ± 0.114 | 0.728 ± 0.074 |
| length-matched triples, teacher forcing | 0.833 ± 0.183 | 0.926 ± 0.084 | 0.833 ± 0.183 | 0.966 ± 0.038 |
Teacher-forced and free-running exact accuracy are identical within every condition; providing correct preceding tokens therefore does not rescue the short-trained model. Scheduled sampling gives only a small, unstable change, whereas exposure to the target length raises exact generation dramatically in every seed. The primary failure is length extrapolation/termination, not classic exposure bias. A large natural run should include long targets and validate by length bucket rather than assume data volume alone will fix stopping behavior.
The audit report now includes a broad affix inventory and per-seed power table
(train>=20, test>=10 valid within-split base/derived pairs), so a future import
reveals before training which prefixes/suffixes support claims. It also exports
MinHash/LSH p99/max bucket sizes, oversized-bucket participation, largest leakage groups,
and records absorbed by groups over 100.
This power report also exposed and fixed a preflight split-allocation bug: assigning
groups by absolute remaining records systematically put most multi-record families in
train. Groups are now allocated by relative remaining capacity. Split sizes stay exact,
while -ly/-ing/-ed receive powered test coverage in most/all current seeds; scarce
un-/-less/-ness remain explicitly underpowered. This is precisely the information
the future 500k audit must surface before an experiment is authorized.
A synthetic 500,000-record infrastructure stress test (not a linguistic experiment)
first rejected the original four-band SimHash implementation (0.508 injected-pair
recall; 1,868 silently oversized buckets). Its disk-backed replacement now produces:
| preflight item | result | decision |
|---|---|---|
| grouped split sizes | 350,000 / 75,000 / 75,000 | GO |
| injected near-duplicate recall | 1.000 | GO |
| silently skipped buckets | 0 | GO |
| maximum / p99 LSH bucket | 73,488 / 30 | monitored |
| max candidates in exact recall route | 4,083 | GO (<10k) |
| SQLite size / peak traced memory | 329 MB / 66 MB | GO |
| runtime | 307 s | GO |
The strict greedy group allocator remains exactly balanced in the simulated heavy-tail
case, but real 500k data must still pass a largest-cluster threshold before training.
Large common-language LSH buckets remain visible (maximum 73,488), but they are never
discarded; length/cardinality filtering bounds candidates before exact verification.
The replacement therefore passes the declared synthetic preflight. Real imported data
must rerun the same recall, candidate-load, disk, and cluster checks rather than inherit
this synthetic GO automatically. Reproduce it with
python scripts/stress_test_data_audit.py.
All results below use directed_gnn (Experiment 1's architecture and, where noted,
its trained checkpoint), 3 seeds, mean ± std unless stated otherwise.
-
Held-out edges (standard test split): see Experiment 1's table above — GNN pooled ROC-AUC 0.621 ± 0.014.
-
Hard-negative-only evaluation: already broken out above per level (
hard_structural0.610 ± 0.024,lexical0.489 ± 0.023) — not hidden behind the pooled number. -
Held-out nodes (inductive split) — 270 nodes (10%) held out entirely from training loss; each keeps 50% of its real edges visible as message-passing context (never used for loss), the other 50% (~1140 "probe" edges/seed) scored only after training, using the same frozen aggregation weights (
experiments/generalization/inductive.py):level probe ROC-AUC easy 0.640 ± 0.023 degree_aware 0.553 ± 0.021 hard_structural 0.577 ± 0.017 lexical 0.479 ± 0.020 pooled 0.579 ± 0.020 Genuine but limited generalization to nodes never seen during training: pooled 0.579 vs. 0.621 transductive — a real drop, not a collapse to chance. This only works because node features are purely structural (see below); a plain per-node embedding table (like the
embedding_dot_productbaseline) has no defined behavior at all for a node it never trained a row for. -
Degree-bucket evaluation (4 train-degree quartiles of the edge's source word, Experiment 1's test edges, Experiment 1's trained checkpoint,
experiments/generalization/degree_bucket.py):degree bucket (low → high) pooled ROC-AUC bucket_0 (lowest) 0.614 ± 0.012 bucket_1 0.642 ± 0.013 bucket_2 0.649 ± 0.018 bucket_3 (highest) 0.689 ± 0.014 A real, monotonic gap: the model is measurably better at reconstructing edges for popular words than for the long tail of low-degree words. Aggregate numbers above over-represent how well this generalizes to rare words.
-
Shuffled-label / randomized-graph sanity check — same architecture, same training procedure, same split methodology, applied to a fresh degree-preserving-configuration-model reshuffling of the real graph each trial (3 trials/seed, 9 total, reusing
graph/null_models.py,experiments/generalization/randomized_graph.py):pooled ROC-AUC real graph 0.621 ± 0.014 randomized graph (9 trials) 0.530 ± 0.021 Performance drops substantially (though not all the way to 0.5) on the randomized graph, supporting that the model is picking up on graph structure specific to this lexicon's actual definitions, not merely degree/density patterns that a degree-preserving shuffle would preserve. The residual ~0.53 (rather than exactly 0.5) is consistent with the model still exploiting some degree signal even on the shuffled graph, since degree is literally its only input feature.
models/gnn.py's DirectedGraphSAGELinkPredictor takes [log1p(out_degree), log1p(in_degree)] as its only per-node input — no learnable per-node ID embedding
table. This is a deliberate trade-off, not an oversight: a shared, node-agnostic
input feature is what makes the same architecture usable, unmodified, for the
inductive generalization test above (a held-out node has no ID-embedding row to fall
back on, but does have degree computed from whatever context edges are visible). The
embedding_dot_product baseline is where a per-node lookup table is deliberately
tested instead, precisely to make this trade-off visible in the results rather than
asserted.
The graph is tiny (2707 nodes, ~12.4k edges). A directed mean-aggregation layer is
~30 lines of torch.index_add-based scatter-mean and is easy to audit end-to-end
(models/gnn.py); pulling in PyG adds a large dependency with its own build/version
surface for no architectural benefit at this scale. PyG would earn its place for a
much larger graph or an architecture that needs its optimized sparse/attention
kernels — neither applies here. In- and out-neighborhoods are aggregated separately
(never symmetrized into an undirected graph), and the final scorer applies distinct
learned projections to a node's "source role" vs. "target role", so
score(u, v) != score(v, u) in general.
This project measures reconstruction of held-out relations in one specific closed
lexicon, detection of synthetic structural corruption, and generalization
beyond directly observed edges/definitions — under multiple difficulty levels, with
null-model and randomized-graph controls. It does not measure, and this document
never claims, that any model "understands" or "learns the meaning of" these words.
Every result above is reported per negative-difficulty / corruption level and per
generalization test separately, including the ones that look bad
(directed_common_neighbor on hard negatives, directed_gnn on lexical negatives,
corrected definition_swap at chance, the real-vs-randomized-graph gap not reaching
all the way to chance) — a model or metric that only looks good in a pooled average is called
out as such above, not hidden behind it.
Scope restrictions honored throughout: no pretrained language models, no external embeddings (GloVe/word2vec/BERT/sentence-transformers/API embeddings), no externally-assigned semantic labels of any kind. All embeddings (node, token, word) are trained from scratch on this dataset alone.
python -m venv .venv && source .venv/bin/activate # or use --without-pip + get-pip.py
# bootstrap if ensurepip is unavailable
pip install -e ".[dev]"
pytest tests/ -v # 111 tests, ~4s
python scripts/build_graph.py # graph stats + null-model sanity check -> results/graph_stats.json
python scripts/run_exp1.py # -> results/exp1_link_prediction/
python scripts/run_exp2.py # -> results/exp2_corruption_detection/
python scripts/run_exp2_encoder_ablation.py # reuses exp2 control; trains the 3 new ablation cells
python scripts/plot_encoder_ablation.py # -> ablation summary + training-curve PNGs
python scripts/run_exp3_known_word.py # known-word ceiling + inductive exposure buckets
python scripts/plot_exp3_known_word.py # -> known-vs-inductive comparison PNG
python scripts/run_exp4_context_encoder.py # char/context inductive encoder ablation
python scripts/plot_exp4_context_encoder.py # -> context encoder summary PNG
python scripts/run_exp5_graph_retrieval.py # directed graph + BCE/pairwise/InfoNCE
python scripts/plot_exp5_graph_retrieval.py # -> graph retrieval summary PNG
python scripts/run_exp6_hard_retrieval.py # hard InfoNCE + gradient diagnostics
python scripts/eval_exp5_full_retrieval.py # full retrieval for saved pure InfoNCE
python scripts/plot_exp6_hard_retrieval.py # -> hard retrieval diagnostic PNG
python scripts/run_exp7_density_curve.py # nested 25/50/75/100% InfoNCE curve
python scripts/plot_exp7_density_curve.py # -> coverage/performance curve PNG
python scripts/run_exp8_evidence_attention.py # character/evidence attention InfoNCE
python scripts/plot_exp8_evidence_attention.py # -> spelling/evidence comparison PNG
python scripts/run_exp9_morphology_audit.py # disjoint-family character retrieval
python scripts/plot_exp9_morphology_audit.py # -> family-split comparison PNG
python scripts/run_exp10_productive_morphology.py # nonce-affix/order audit (reuses Exp. 9 checkpoints)
python scripts/plot_exp10_productive_morphology.py # -> productive morphology audit PNG
python scripts/run_exp11_character_cnn.py # order-sensitive CNN on family splits
python scripts/plot_exp11_character_cnn.py # -> CNN/retrieval + nonce-order PNG
python scripts/run_exp12_order_contrast.py # bidirectional matched-order InfoNCE
python scripts/plot_exp12_order_contrast.py # -> real/nonce order-control PNG
python scripts/run_exp13_semantic_affix.py # frozen data-derived semantic affix audit
python scripts/plot_exp13_semantic_affix.py # -> semantic prototype/control PNG
python scripts/run_exp14_compositional_reconstruction.py # base + affix-delta retrieval
python scripts/plot_exp14_compositional_reconstruction.py # -> compositional retrieval PNG
python scripts/run_exp15_definition_intervention.py # causal definition-assignment null
python scripts/plot_exp15_definition_intervention.py # -> intervention summary PNG
python scripts/run_exp16_multistep_composition.py # controlled held-out operation chains
python scripts/run_exp17_definition_generation.py # free compositional definitions
python scripts/run_exp18_controlled_scaling.py # synthetic coverage/scale curve
python scripts/run_exp19_controlled_understanding.py # multi-task controlled battery
python scripts/plot_exp16_to_19_reasoning_ladder.py # -> combined reasoning-ladder PNG
python scripts/audit_lexicon.py # canonical records + strict leakage-safe splits
python scripts/run_exp20_decoder_diagnosis.py # length vs exposure-bias ablation
python scripts/stress_test_data_audit.py # synthetic 500k infrastructure preflight
python scripts/run_generalization.py # requires run_exp1.py to have run first (reuses its checkpoints/splits)
# or: python scripts/run_all.pyEach script accepts --seeds 0 1 2 ... to override the config's seed list. All
configs are YAML under experiments/*/config.yaml — no hyperparameters are hardcoded
in the training code. Every run writes seeded, deterministic split artifacts
(data/processed/splits/), model checkpoints (results/checkpoints/), and a JSON
result file stamped with seed, config, git commit hash, and timestamp
(training/utils.run_metadata). GPU is used automatically via
torch.cuda.is_available(); the environment this was built in only had a CPU-only
PyTorch wheel available (network constraints — see commit history), so all reported
numbers above are CPU runs; nothing in the code path is GPU-specific.
Training, validation, and test corruption/negative samplers use separate deterministic random streams. Every Experiment 1 model is evaluated against the same sampled candidate pairs; test candidates therefore do not depend on model order, early-stopping epoch, or how many random samples a preceding training run consumed.
Two hyperparameter choices were tuned by measurement, not guessed, and are documented
inline in the relevant config.yaml: the embedding baseline's learning rate/patience
(the default lr=0.01, patience=20 left it stuck at chance under full-batch
gradient descent — sparse per-node rows only get gradient when their node appears in
a batch, so it converges much slower than the GNN's shared weights) and the GNN's/
corruption model's patience (too-short patience was measured to trip early stopping
before validation performance plateaus).