PoC: deduplicate structurally identical structs in the provider generator

[jsteinich]

Proposal: #389 — this PR is the proof-of-concept code referenced there. Read #389 first for the full investigation; this PR covers the implementation only.

Draft / proof of concept — not for merge. Opened to show the code behind the proposal. Behind CDKTN_STRUCT_DEDUP=1; unset, generated output is byte-identical to today.

What this is

The generator names a struct after the path that reaches it, so one block shape reachable by many paths is emitted once per path — along with its mapper functions and OutputReference/List classes, which are 84% of generated bytes.

On datadog 4.18, 95.5% of generated interfaces are structurally identical duplicates. The largest equivalence class is 364 byte-identical interfaces.

This is what OOM-kills package-go: not the Node heap ceiling (tuned three times), but the go build verification inside jsii-pacmak, which needs ~18 GB for a single generated package on a runner with ~24 GB usable. That memory is not governed by NODE_OPTIONS at all.

The duplication is an artifact of schema serialization

terraform providers schema -json inlines every call site and discards sharing the provider author wrote by hand:

getComputeSchema()                              <- 1 Go function
  ^ called 2x inside
getApmLogNetworkRumSecurityAuditQuerySchema()   <- 48 call sites
  + getMetricQuerySchema (11), getFormulaQuerySchema (10),
    getProcessQuerySchema (10), getApmStatsQuerySchema (2)
  v
364 generated TypeScript interfaces

Cross-resource duplication has the same origin: resource_datadog_powerpack.go calls getNonGroupWidgetSchema() defined in resource_datadog_dashboard.go. Dedup recovers structure the provider actually expressed — it is not merging things that merely look alike.

detectAttributeLoops is not broken

It threads knownStructs down each DFS branch, so it merges only when a shape reappears among its own ancestors (true recursion). Sibling branches are never compared and each top-level attribute restarts from {}. The 364-member class is ...ApmQueryComputeQuery vs ...LogQueryComputeQuery — siblings, invisible to an ancestor-only check by construction. It works as designed; its scope cannot see this.

Equality must be a full recursive signature

This pass deliberately does not reuse getAttributeIdentifier. That comparison uses attribute names plus one level of nesting (with an in-code caveat that it is an approximation). Safe for the ancestor-only case; applied resource-wide it yields 199 classes where an exact recursive hash yields 200 — wrongly merging DashboardV2WidgetCohortDefinition with DashboardV2WidgetGroupDefinitionWidgetSloListDefinition, which share a shallow shape and diverge deeper. That would emit incorrect bindings.

Measured — datadog 4.18, end to end

Stage	Baseline	With dedup
Generated TS	99.6 MB	12.3 MB (-87.7%)
Interfaces	13,220	1,491
tsc	0 errors, peak 4,674 MB	0 errors, peak 744 MB
jsii	—	0 errors, assembly 22 MB
jsii-pacmak --target go	—	rc=0, 8,110 files, 56.5 MB
go build ./... (GOMAXPROCS=8, cold)	~18 GB, OOM-killed	925 MB, 11.1 s, exit 0

925 MB fits a standard 7 GB runner at full parallelism, with the compilation check kept.

Why this is not ready to merge

1. Naming is a placeholder. The canonical-name rule (shortest name, lexicographic tie-break) is order-independent but still not stable across provider releases — 0.2-0.8% of shapes renamed on datadog 4.17 -> 4.18, because the elected member can leave the equivalence class. The intended fix is a checked-in signature -> name registry with incumbents winning collisions; not implemented here.
2. Breaking change. 13,220 -> 1,491 exported names on datadog. jsii has no type aliases, and interface A extends B {} would not help since it retains the per-name classes and functions that are 84% of the bytes. Major bump per provider.
3. No tests yet.

Scope: per-resource, not provider-wide (tested)

	datadog	aws	awscc
Shapes per-resource -> provider-wide	1,257 -> 879	6,544 -> 4,758	12,698 -> 8,763
Name length	30 -> 25	37 -> 30	41 -> 57
Resolve at 1 segment	75% -> 36%	94% -> 43%	95% -> 2%
Rename rate 4.17->4.18	0.56% -> 1.80%	—	—

Provider-wide is never clearly better and is actively bad for awscc (machine-generated CloudFormation schemas give every resource the same generic block names).

Fleet impact — datadog is an outlier

Provider	Interfaces	After dedup	Delivered	Provider-wide ceiling
datadog	13,220	1,491	88.7%	90.3%
aws	12,188	8,923	26.8%	54.3%
awscc	21,026	16,749	20.3%	73.2%

Most providers' duplication is across resources, which per-resource scope does not capture. Six providers (acme, cfncompat, external, http, null, time) have 0% redundancy — a no-op with no API impact.

Caveats

• Prototyped against generator 0.21 (baseline reproduced published 0.24 output exactly: 99.6 MB / 346 files), then ported here; typechecks clean.
• dashboardv2 compile figures measured on a Windows host with 32 GB, not Linux CI.
• The ~18 GB for 4.18 is the measured 4.17 figure scaled by output growth — not directly measured.
• Go compile impact for aws/awscc after dedup not measured.

🤖 Generated with Claude Code

[jsteinich]

Context from the #387 investigation, for whoever extends this PoC to group scope.

Summary: the recommendation has moved from per-resource to per-group (service-scoped) dedup. Measured on the published Go trees of both providers, with references resolved within a package and signatures compared across the scope under test:

scope	aws structs → shapes	awscc structs → shapes
per-resource	12,403 → 8,073 (34.9%)	22,051 → 11,985 (45.6%)
per-group	12,403 → 6,475 (47.8%)	22,051 → 7,397 (66.5%)
provider-wide	12,403 → 5,496 (55.7%)	22,051 → 5,836 (73.5%)

Per-group captures ~90% of provider-wide's byte benefit (aws 209.3 of 233.0 MB reclaimed; awscc 360.8 of 390.6 MB) without the two costs #389 rightly rejected provider-wide for: the naming blowup — for awscc, provider-wide made names longer, with 97% embedding a resource name — and a shared-types module that every module in a Go split would have to depend on. Under per-group scope, merged types stay inside the group that owns them.

It also fixes the one thing that made service grouping look expensive. Regrouping jsii submodules per service makes each group a single Go package, and Go compiles a package as a unit — which would have taken the largest package to 57,166,831 B (~6.3 GB compile, using #389's measured RSS-per-MB). With per-group dedup applied it lands at 20,699,765 B (~2.3 GB) for aws and 21,360,127 B for awscc — below today's worst ungrouped package, aws/lexv2modelsintent at 48,722,416 B (~5.4 GB). Grouping only hurts compile cost if done without dedup.

Treat those yields as a ceiling, not a target

This PoC's signature() encodes cycles as @rec:${depth}, recording how far up the cycle points. My measurement collapses every cycle to a flat <rec> token, which is looser and will over-merge. That is very likely why my numbers run above #389's for the same providers (aws 34.9% vs 26.8% per-resource). Expect this PoC to land below the figures above — that's my measurement being imprecise, not a regression here. The ordering between scopes should hold.

Per-group is not a scope parameter on this pass

Worth being explicit, because it changes the size of the job. This PoC merges inside parseResource, one resource at a time, so a merged type stays in its own submodule and no new boundary appears. Group scope needs:

• deferring the merge until every resource in the group is parsed — a pipeline change, not a flag
• a group-level jsii submodule to hold the shared struct, i.e. provider-generator.ts and the Scope/namespace machinery, not just the parse pass
• cross-submodule references in the emitted TS
• names unique within the group

And it is cross-language by construction: the assembly has 2,416 jsii submodules ↔ 2,416 Go packages, every type's FQN embeds its submodule (@cdktn/provider-aws.<submodule>.<Type>), and zero types live in the root namespace. There is no way to introduce a boundary only Go sees. (Dedup at any scope is already all-language — Go registers types against assembly FQNs, so types can't be merged in the Go output alone.)

Practically this means per-group dedup and cross-language service regrouping are the same decision. Per-resource dedup remains the fallback if regrouping isn't wanted.

Grouping rule — free for both providers

aws — terraform-provider-aws, names/data/names_data.hcl. For each service "x" {} and nested sub_service "y" {} block, read resource_prefix { actual, correct }; match a resource against both patterns anchored at ^, longest match wins; a sub_service maps to its split_package parent when present. Two gotchas:

• HCL escapes backslashes, so \\b in the file is the regex \b.
• You must still try correct when actual is present, or kinesis mis-resolves.

Yields 250 groups in use (373 defined). Sanity checks that must pass: aws_instance / aws_ami / aws_eip / aws_vpc / aws_ebs_volume → ec2; aws_db_instance → rds; aws_cloudwatch_log_group → logs; aws_s3control_bucket → s3control, not s3.

Stability is measured, not assumed: across all 354 revisions of that file from 2024-06-11 to 2026-08-18, 0 of 1,721 resources ever changed group, and growth was purely additive (+34 groups). The structural reason is that the Terraform resource name contains its service prefix, so a reassignment would require renaming the resource.

awscc — no data file needed and none exists. Names are generated from CFN types, so the service is the second underscore segment (awscc_s3_bucket → s3); of 2,709 unique resources, 0 lack one. 287 groups.

Operationally, vendor a snapshot of names_data.hcl rather than fetching at generation time, with a fallback (a resource's own prefix becomes its own group) and a CI check that flags unmapped resources — so a new upstream service can't silently mis-group against a stale snapshot.

Sequencing note

#389 argues the name registry must land before anything ships, and that stands. But group scope changes the registry's design inputs — collision counts and rename rates are measured per-scope — so it's worth having the grouped prototype emit those numbers as it goes rather than re-deriving them afterwards.

Full measurements, module anatomy, growth rates (aws 0.469 MB/day, awscc 1.60 MB/day) and why awscc forces a module split regardless of trimming are in two comments on #387. #441 is the interim non-breaking Go lever.

🤖 Generated with Claude Code

[jsteinich]

Scope note: this PoC is per-resource by construction

Group (service) scope was not in the original comparison, and measurement since (see #389 and the module-size work in #387) shows it beats per-resource on both size and naming for the two largest providers:

	per-resource	per-group	provider-wide
aws — shapes / name length	6,544 / 37	5,371 / 28	4,758 / 30
awscc — shapes / name length	12,698 / 41	9,662 / 29	8,763 / 58

Per-group produces the shortest names of all three scopes, so the naming objection that ruled out provider-wide does not apply to it.

This branch cannot express that. The merge runs inside parseResource, one resource at a time, so it is structurally per-resource — group scope is not a flag on this pass. It needs:

• deferring the merge until every resource in a group is parsed,
• a group-level jsii submodule to own the shared struct (provider-generator.ts plus the Scope/namespace machinery, not just the parse pass),
• cross-submodule references in the emitted TS,
• names unique within the group rather than within the resource.

That is emit-layer work and cross-language by construction, since jsii submodules surface in all five targets.

What stays valid here is everything the PoC was opened to show: the duplication exists and is large (95.5% on datadog 4.18), the recursive signature is the correct equality test and the one-level getAttributeIdentifier is not, and the end-to-end numbers (go build ./... 925 MB / 11.1 s / exit 0 versus ~18 GB) hold — datadog's win comes almost entirely from intra-resource duplication, where per-resource already captures 88.7% of a 90.3% ceiling.

So this remains a useful demonstration of the mechanism and a working measurement harness, but it should not be read as the recommended scope for aws/awscc. Whoever picks up the implementation should plan for group scope in the emit layer rather than extending this pass.

Also worth carrying over: the grouping rule needs a tie-break. aws_ebs_volume matches service "ebs" and sub_service "ec2ebs" at the same length; preferring actual over correct resolves it correctly to ec2. Details in #389.